Supporting Julia 1.0+

[henry2004y]

Hi,

Is there any plan to continue supporting Julia version 1.0+?

[BobPortmann]

I have not used this code for years, as I have not really needed it. I have no plans to keep working on it. However, it should not be too hard to update it for 1.0+ and I would be happy to review any changes. But note that there is a serious issue with focus when using IDL graphics from julia. The IDL widgets do not get any clicks or other interactions. I just want to point this out in case using IDL graphics is your primary goal in using IDL from Julia.

[henry2004y]

Thanks for the reply! Actually my goal is not to use the graphics in IDL, but to gradually convince my advisor that the legacy IDL scripts can be transferred to Julia with ease. I thought it would be interesting to get this working in Julia 1.0+, and I have already made some commits.

When I attempted to use the rpc yesterday, first I just typed

ccall((:IDL_RPCInit, idlrpc), Ptr{Nothing}, (Clong, Ptr{UInt8}), 0, C_NULL)

with the idlrpc hard-coded on my Mac, it showed me

rpc: remote system error - connection refused

Any idea about that error message?

[BobPortmann]

You need to start idlrpc before calling ccall, either by running idlrpc in another window at a shell prompt (assuming its on your path) or by spawning it inside julia. The old code did the latter by calling spawn(`idlrpc`) (look in idlrpc.jl) but the spawn command changed and I'm not sure what the new function is now. So try the former.

[BobPortmann]

Okay, I found the new command. This works for me (you may have to change the path and idlrpc needs to be on the shell path):

Crow ~> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> const idlrpc = "/Applications/harris/idl87/bin/bin.darwin.x86_64/libidl_rpc"
"/Applications/harris/idl87/bin/bin.darwin.x86_64/libidl_rpc"

julia> run(`idlrpc`, wait=false)
Process(`idlrpc`, ProcessRunning)

julia> ccall((:IDL_RPCInit, idlrpc), Ptr{Nothing}, (Clong, Ptr{UInt8}), 0, C_NULL)
Ptr{Nothing} @0x00007fa0fd97e5a0

julia> 

[henry2004y]

For me, the following parts work so far:

IDL_LIB_DIR = "/Applications/exelis/idl85/bin/bin.darwin.x86_64"
const libidl_rpc = joinpath(IDL_LIB_DIR,"libidl_rpc.dylib")
const idlrpc = "/Applications/exelis/idl85/bin/idlrpc"
run(`$idlrpc`, wait=false)
ccall((:IDL_RPCInit, libidl_rpc), Ptr{Nothing}, (Clong, Ptr{UInt8}), 0, C_NULL)

The idlrpc executable and libidl_rpc are actually under different directories. I'll continue to see what happens next.

---

Now I managed to call IDL through RPC! Things still need to be cleaned up.

Several things are bothering me at the moment:

1. Regular expression section for line continuations. Many functions have changed so I need to reimplement this part. However, as I am not very familiar with regular expressions, this is giving me headache. If I understand the code correctly, for example, we want

line = """
a = 1 \$
+1
b = '1 \$';; comments
"""

to be correctly passed to IDL. I got stuck with ruling out the [;|$] inside quotes. If I ignore that, simply

line = """
a = 1 \$
+1
b = '1 +1';;comments
"""
line = replace(line, r"(;|\$).*(\n|$)"=>"")

would work in Julia 1.3.

2. Something interesting in using unsafe_wrap in common-funcs.jl, get_var():

parr = reinterpret(Ptr{IDL_Array}, convert(Int, var.buf))
idl_arr = unsafe_load(parr)
jl_t = jl_type(var.vtype)
pdata = reinterpret(Ptr{jl_t}, idl_arr.data)
 
arr = unsafe_wrap(Array, pdata, dims(idl_arr.dim, idl_arr.n_dim))
#=
arr = Array{jl_t}(undef, dims(idl_arr.dim, idl_arr.n_dim))
for i = 1:idl_arr.n_elts
   arr[i] = unsafe_load(pdata, i)
end
=#
# If you don't copy, the pointer will be freed!
return copy(arr)

In the above code, unsafe_wrap actually works (simple check by @show arr). However, if you return arr directly, somehow the pointer is no longer valid...

julia> line = """
       1+1
       """
"1+1\n"

julia> execute(line)
1+1
 ^
% Syntax error.

This is not working probably because currently we cannot capture expressions that do not assign to any variable or function. However, this shouldn't matter too much.

4. REPL
   I am not familiar with this part at all. Now I think LineEdit comes from REPL module. Also, just as the init() function in IDLRPC.jl, in julia 1.0+ it is not allowed to execute functions inside module except __init__(), which is called once and only once during the initialization of the package.
   Somehow magically after some small fix it just works! However, I need to manually call init_repl() after using IDL.

5. Callable
   This part is not working on Mac:

using Libdl
const idlcall = "/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib"
Libdl.dlopen(idlcall)

Error message:

ERROR: could not load library "/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib"
dlopen(/Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib, 1): Library not loaded: libMesaGLU6_2.dylib
  Referenced from: /Applications/exelis/idl85/bin/bin.darwin.x86_64/libidl.dylib
  Reason: image not found

6. A very strange error when calling init_repl() in __init()

This works perfectly fine if I just say using IDL. However, if I move to the package manager by typing ] and then type test, it showed:

(IDL) pkg> test
   Testing IDL
 Resolving package versions...
ERROR: LoadError: InitError: UndefVarError: active_repl not defined
Stacktrace:
 [1] idl_repl() at /Users/hyzhou/Documents/Computer/Julia/IDL/src/IDLREPL.jl:10
 [2] __init__() at /Users/hyzhou/Documents/Computer/Julia/IDL/src/IDL.jl:41
 [3] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:692
 [4] _require_from_serialized(::String) at ./loading.jl:743
 [5] _require(::Base.PkgId) at ./loading.jl:1034
 [6] require(::Base.PkgId) at ./loading.jl:922
 [7] require(::Module, ::Symbol) at ./loading.jl:917
 [8] include at ./boot.jl:328 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1105
 [10] include(::Module, ::String) at ./Base.jl:31
 [11] include(::String) at ./client.jl:424
 [12] top-level scope at none:6
during initialization of module IDL
in expression starting at /Users/hyzhou/Documents/Computer/Julia/IDL/test/runtests.jl:1
ERROR: Package IDL errored during testing

I don't quite understand why. I asked the same question on discourse.

---

Summary so far:
I have successfully made RPC and REPL working on Mac, but no luck in Callable. I have created new Manifest.toml and Project.toml file, and modified README.md. For my own usage, this is probably enough. Also, I changed the package name from IDLCall.jl to IDL.jl. I will send a PR; review it if you have any free time!

[henry2004y]

#13