SCS_GPU: build libscsgpuindir against CUDA_full_jll #1294

kalmarek · 2020-07-07T16:15:07Z

Builds only libscsgpuindir against CUDA_full_jll;
this simplified the build script (only the GPU part is built) and I bit the bullet and tested the build locally ;)

this moves #1289 into a separate package following advice of @giordano;

I limited the platforms only to the supported by CUDA; here is the output of ldd on BB compiled library:

$ ldd ./products/lib/libscsgpuindir.so
        linux-vdso.so.1 (0x00007ffe2eeb9000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f985dc8f000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f985dc84000)
        libopenblas64_.so => not found
        libcudart.so.11.0 => not found
        libcublas.so.11 => not found
        libcusparse.so.11 => not found
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f985dabb000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f985e03a000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f985da99000)

by comparison this is what I get compiling on an external machine:

$ ldd ./out/libscsgpuindir.so 
        linux-vdso.so.1 (0x00007ffd3377f000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f4f63b1b000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f4f63b10000)
        libopenblas64_.so.0 => not found
        libcudart.so.10.2 => /opt/cuda/lib64/libcudart.so.10.2 (0x00007f4f63892000)
        libcublas.so.10 => /opt/cuda/lib64/libcublas.so.10 (0x00007f4f5f5dc000)
        libcusparse.so.10 => /opt/cuda/lib64/libcusparse.so.10 (0x00007f4f57976000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f4f577ad000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4f63cbb000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f4f5778b000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f4f57785000)
        libcublasLt.so.10 => /opt/cuda/lib64/libcublasLt.so.10 (0x00007f4f558f2000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f4f55715000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f4f556fb000)

What I understand any CUDA > 9.0 (or maybe even earlier) would do;

@maleadt As I understand what you said in the other thread, witch such compiled SCS_GPU one can still import SCS_GPU_jll without CUDA_jll etc. What would be the optimal way for discovery CUDA at runtime?

At the moment we do the following: if SCS is compiled from source then possibly add libscsgpuindir (libscsgpu is the old name):
https://github.com/jump-dev/SCS.jl/blob/672b6711e924ca1af520b48db73da217bb00e6ba/deps/build.jl#L63

Then at precompile time we add it to a const available_solvers:
https://github.com/jump-dev/SCS.jl/blob/672b6711e924ca1af520b48db73da217bb00e6ba/src/c_wrapper.jl#L90

Would it be feasible to check for CUDA at __init__ and populate available_solvers there?

S/SCS_GPU/build_tarballs.jl

giordano · 2020-07-07T16:18:24Z

S/SCS_GPU/build_tarballs.jl

+platforms = [
+    Linux(:x86_64),
+    Windows(:x86_64),
+    MacOS(:x86_64),
+]


Following a discussion we had with @maleadt a few weeks ago my understanding is that this is going to work only for Linux

I see, I've taken my clues from MAGMA:

Yggdrasil/M/MAGMA/build_tarballs.jl

Line 37 in 54c9c4b

platforms = [

It depends on CUDA_jll and builds for all of those platforms... [email protected] builds for those as well:

Yggdrasil/C/CUDA/[email protected]/build_tarballs.jl

Line 189 in 816eae9

build_tarballs(ARGS, name, version, [], script,

@maleadt could you chip in?

but as You have predicted Linux builds fine, MacOSX and Windows fail 😆

I don't think magma ever worked at all

Co-authored-by: Mosè Giordano <[email protected]>

maleadt · 2020-07-09T10:31:26Z

Seeing how this library links against very specific libraries, e.g.libcusparse.so.11, I'm thinking whether it wouldn't be better to create explicit CUDA11_jll packages. The reason being that it's perfectly possible to have different applications use different versions of the CUDA toolkit, e.g. SCS_GPU using CUDA 11 (libcusparse.so.11) while another uses CUDA 10 / libcusparse.so.10. Since the soversion changes every time, it's impossible to have a single CUDA_jll version-resolved by Pkg because it needs to match exactly anyhow. Thoughts, @giordano?

Regardless, for this build you better use the oldest version of CUDA that is compatible, because for CUDA 11 you need NVIDIA driver 450+ (or so) which is 0% of the users currently.

giordano · 2020-07-09T10:50:04Z

Seeing how this library links against very specific libraries, e.g.libcusparse.so.11, I'm thinking whether it wouldn't be better to create explicit CUDA11_jll packages. The reason being that it's perfectly possible to have different applications use different versions of the CUDA toolkit, e.g. SCS_GPU using CUDA 11 (libcusparse.so.11) while another uses CUDA 10 / libcusparse.so.10. Since the soversion changes every time, it's impossible to have a single CUDA_jll version-resolved by Pkg because it needs to match exactly anyhow. Thoughts, @giordano?

We already have some infrastructure in place to make something like this happen (but we aren't really there yet) 😉 pinging @staticfloat

kalmarek · 2020-07-09T11:43:55Z

@maleadt I fixed cuda to 9.0;

What could be the design pattern for conditional use of in SCS?

maleadt · 2020-07-09T13:16:36Z

What could be the design pattern for conditional use of in SCS?

Check if libcuda.so is available (Libdl.dlopen(;dont_error=true)), and if so, using SCS_GPU_jll. Because JLLs dlopen their libraries during __init__, you can't unconditionally import, sadly. Another potential area of improvement in BinaryBuilder.jl.

If you want to use this together with CUDA.jl, that's tricky, because CUDA.jl does its own artifact selection (the most recent compatible one for your driver), vs. 9.0 you've selected here. So that could cause multiple libcublas libraries to get loaded, and I don't think that will work very nicely. If CUDA.jl would switch to ccall-by-soname, it would reuse whatever libcublas was available already, but that creates another whole mess of potential compatibility issues. Nothing is ever straightforward in CUDA land...

kalmarek · 2020-07-10T10:51:20Z

Nothing is ever straightforward in CUDA land...

:D I see; I made the necessary changes to SCS.jl locally. Is there an easy way to test it with SCS_GPU_jll locally as well? e.g. how to install SCS_GPU_jll artifact from path?

giordano · 2020-07-10T11:01:28Z

Is there an easy way to test it with SCS_GPU_jll locally as well? e.g. how to install SCS_GPU_jll artifact from path?

https://juliapackaging.github.io/BinaryBuilder.jl/dev/FAQ/#Can-I-publish-a-JLL-package-locally-without-going-through-Yggdrasil?-1

kalmarek · 2020-07-23T22:35:23Z

I'm trying to figure out a way of detecting CUDA libraries at runtime, but I can't really figure how.

import SCS_jll
const indirect = SCS_jll.libscsindir
const direct = SCS_jll.libscsdir
if Sys.islinux() && Libdl.dlopen("libcublas.$(Libdl.dlext).9.0"; throw_error=false) !== nothing
    import SCS_GPU_jll
    const gpuindirect = SCS_jll.libscsgpuindir
end

This code will be run at precompile time, so libcublas from CUDA_jll will not be available (as it's not in the deps).
On the other hand imports need to be done at module level, so I can't do it during __init() as well... @maleadt am I chasing my own tail? :)

kalmarek · 2020-08-20T21:38:34Z

@maleadt @giordano I couldn't come up with anything better than this:

don't depend on CUDA_jll
add SCS_GPU_jll to deps of SCS.jl
ask users to perform

using Pkg; pkg"add [email protected]"
using CUDA_jll
using SCS
SCS.eval(:(
    import SCS_GPU_jll;
    const gpuindirect = SCS_GPU_jll.libscsgpuindir;
    GpuIndirectSolver in available_solvers || push!(available_solvers, GpuIndirectSolver);
    include("src/c_wrapper.jl"); # to @eval the ccalls depending on gpuindirect 
    ))

see jump-dev/SCS.jl#187 (comment)

If that's good enough, I'd ask to merge SCS_GPU here.

maleadt · 2020-08-21T05:22:03Z

Can't you @requires that code to automatically activate from SCS when SCS_GPU_jll is available? But yeah, the run-time part here is a bit messy right now, and needs some extensions to Julia's/BB's notion of a 'platform' before we can truly fix this.

kalmarek · 2020-08-21T08:02:37Z

@maleadt the problem is that in src/c_wrapper.jl we metaprogram the necessary ccalls based on the availability of scslibs.
https://github.com/jump-dev/SCS.jl/blob/7e56873ca93299e5e8cd5ecc7f52a4c9f46392bc/src/c_wrapper.jl#L96

If I @require CUDA_jll then it's not available at pre-compile, hence no appropriate methods are generated even when CUDA_jll is present at using SCS. I tried several things, but couldn't solve this conundrum ;D I'd be happy to take any new idea though: this SCS.eval is ugly as hell ;)

maleadt · 2020-08-21T09:15:50Z

Ah, yes, that's not ideal. With @runtime_ccall from https://github.com/maleadt/LLVM.jl/blob/c447e61110373502d640335ac89ae971cd844279/src/util.jl#L70-L113, or the upcoming JuliaLang/julia#37123, you can have the ccall target be dynamic, but then you'll still need to generate all methods upfront.

kalmarek · 2020-08-26T09:38:15Z

with dynamic ccall there is no need to meta-program this part, but this will be available only with julia-1.6 I guess; At the moment SCS supports julia-1.0, but maybe the code for gpu could be supported only with dynamic ccall...

kalmarek · 2020-09-03T16:02:44Z

@maleadt apparently I needed to go for holiday to figure out the solution ;-)

The key point is that one can import SCS_GPU_jll inside the @require block!

    function __init__()
        @require CUDA_jll="e9e359dc-d701-5aa8-82ae-09bbf812ea83" include("c_wrapper_gpu.jl")
    end

and the c_wrapper_gpu.jl contains

if haskey(ENV,"JULIA_SCS_LIBRARY_PATH")
    @isdefined(libscsgpuindir) && push!(available_solvers, GpuIndirectSolver)
else
    import SCS_GPU_jll
    const gpuindirect = SCS_GPU_jll.libscsgpuindir
    push!(available_solvers, GpuIndirectSolver)
end
[ ... ]# other gpu specific code

see recent changes in jump-dev/SCS.jl#187
I always thought that using/import must be done on the top level ?!

Anyway:

 ~/.julia/dev/SCS   scs_gpu_jll  julia --project=. 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using CUDA_jll
[NVBLAS] NVBLAS_CONFIG_FILE environment variable is NOT set : relying on default config filename 'nvblas.conf'
[NVBLAS] Cannot open default config file 'nvblas.conf'
[NVBLAS] Config parsed
[NVBLAS] CPU Blas library need to be provided

julia> using SCS
[ Info: Precompiling SCS [c946c3f1-0d1f-5ce8-9dea-7daa1f7e2d13]

julia> SCS.available_solvers
3-element Array{DataType,1}:
 SCS.DirectSolver     
 SCS.IndirectSolver   
 SCS.GpuIndirectSolver

julia> SCS.gpuindirect
"libscsgpuindir.so"

 ~/.julia/dev/SCS   scs_gpu_jll  julia --project=. 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using SCS

julia> SCS.available_solvers
2-element Array{DataType,1}:
 SCS.DirectSolver  
 SCS.IndirectSolver

kalmarek · 2020-09-03T16:04:04Z

@giordano if there are no objections, I'd like to merge it

build libscsgpuindir against CUDA_full_jll

a1642b9

kalmarek mentioned this pull request Jul 7, 2020

build libscsindirgpu.so against CUDA_full_jll #1289

Closed

kalmarek changed the title ~~build libscsgpuindir against CUDA_full_jll~~ SCS_GPU: build libscsgpuindir against CUDA_full_jll Jul 7, 2020

giordano reviewed Jul 7, 2020

View reviewed changes

kalmarek and others added 3 commits July 7, 2020 18:52

use libdir instead of ${prefix}/lib

aa88f7a

Co-authored-by: Mosè Giordano <[email protected]>

change name to SCS_GPU

9c5b982

only Linux CUDA is supported

fbcac16

fix CUDA_full_jll to v9.0

e938470

kalmarek mentioned this pull request Jul 13, 2020

add SCS_GPU_jll jump-dev/SCS.jl#187

Closed

update SCS to 2.1.2

d5f7530

giordano closed this Sep 3, 2020

giordano reopened this Sep 3, 2020

giordano merged commit 904ee24 into JuliaPackaging:master Sep 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SCS_GPU: build libscsgpuindir against CUDA_full_jll #1294

SCS_GPU: build libscsgpuindir against CUDA_full_jll #1294

kalmarek commented Jul 7, 2020

giordano Jul 7, 2020

kalmarek Jul 7, 2020

kalmarek Jul 7, 2020

giordano Jul 7, 2020 •

edited

Loading

maleadt commented Jul 9, 2020

giordano commented Jul 9, 2020

kalmarek commented Jul 9, 2020

maleadt commented Jul 9, 2020

kalmarek commented Jul 10, 2020

giordano commented Jul 10, 2020 •

edited

Loading

kalmarek commented Jul 23, 2020 •

edited

Loading

kalmarek commented Aug 20, 2020

maleadt commented Aug 21, 2020

kalmarek commented Aug 21, 2020

maleadt commented Aug 21, 2020

kalmarek commented Aug 26, 2020

kalmarek commented Sep 3, 2020

kalmarek commented Sep 3, 2020

SCS_GPU: build libscsgpuindir against CUDA_full_jll #1294

SCS_GPU: build libscsgpuindir against CUDA_full_jll #1294

Conversation

kalmarek commented Jul 7, 2020

giordano Jul 7, 2020

Choose a reason for hiding this comment

kalmarek Jul 7, 2020

Choose a reason for hiding this comment

kalmarek Jul 7, 2020

Choose a reason for hiding this comment

giordano Jul 7, 2020 • edited Loading

Choose a reason for hiding this comment

maleadt commented Jul 9, 2020

giordano commented Jul 9, 2020

kalmarek commented Jul 9, 2020

maleadt commented Jul 9, 2020

kalmarek commented Jul 10, 2020

giordano commented Jul 10, 2020 • edited Loading

kalmarek commented Jul 23, 2020 • edited Loading

kalmarek commented Aug 20, 2020

maleadt commented Aug 21, 2020

kalmarek commented Aug 21, 2020

maleadt commented Aug 21, 2020

kalmarek commented Aug 26, 2020

kalmarek commented Sep 3, 2020

kalmarek commented Sep 3, 2020

giordano Jul 7, 2020 •

edited

Loading

giordano commented Jul 10, 2020 •

edited

Loading

kalmarek commented Jul 23, 2020 •

edited

Loading