Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: generator expressions #14848

Merged
merged 3 commits into from
Feb 19, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ CORE_SRCS := $(addprefix $(JULIAHOME)/, \
base/dict.jl \
base/error.jl \
base/essentials.jl \
base/generator.jl \
base/expr.jl \
base/functors.jl \
base/hashing.jl \
Expand Down
3 changes: 3 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ Julia v0.5.0 Release Notes
New language features
---------------------

* Generator expressions, e.g. `f(i) for i in 1:n` (#4470). This returns an iterator
that computes the specified values on demand.

* Macro expander functions are now generic, so macros can have multiple definitions
(e.g. for different numbers of arguments, or optional arguments) ([#8846], [#9627]).
However note that the argument types refer to the syntax tree representation, and not
Expand Down
2 changes: 1 addition & 1 deletion base/abstractarray.jl
Original file line number Diff line number Diff line change
Expand Up @@ -1226,7 +1226,7 @@ function map!{F}(f::F, dest::AbstractArray, A::AbstractArray)
return dest
end

function map_to!{T,F}(f::F, offs, st, dest::AbstractArray{T}, A::AbstractArray)
function map_to!{T,F}(f::F, offs, st, dest::AbstractArray{T}, A)
# map to dest array, checking the type of each result. if a result does not
# match, widen the result type and re-dispatch.
i = offs
Expand Down
1 change: 1 addition & 0 deletions base/boot.jl
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,7 @@ unsafe_convert{T}(::Type{T}, x::T) = x
(::Type{Array{T}}){T}(m::Int, n::Int, o::Int) = Array{T,3}(m, n, o)

# TODO: possibly turn these into deprecations
Array{T,N}(::Type{T}, d::NTuple{N,Int}) = Array{T}(d)
Array{T}(::Type{T}, d::Int...) = Array{T}(d)
Array{T}(::Type{T}, m::Int) = Array{T,1}(m)
Array{T}(::Type{T}, m::Int,n::Int) = Array{T,2}(m,n)
Expand Down
1 change: 1 addition & 0 deletions base/coreimg.jl
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ macro doc(str, def) Expr(:escape, def) end

## Load essential files and libraries
include("essentials.jl")
include("generator.jl")
include("reflection.jl")
include("options.jl")

Expand Down
21 changes: 21 additions & 0 deletions base/generator.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
"""
Generator(f, iter)

Given a function `f` and an iterator `iter`, construct an iterator that yields
the values of `f` applied to the elements of `iter`.
The syntax `f(x) for x in iter` is syntax for constructing an instance of this
type.
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you need to add this signature to the appropriate manual section and then run genstdlib.jl to copy this text over

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't exported, so it would be as Base.Generator

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I wonder about the name of this type. The syntax for it is definitely called a "generator expression" (at least in python), but in python this produces a generator object, which is somewhat similar to a Task, while we return more of a MapIterator. I guess best not to export it for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw this as the equivalent of python's itertools.imap, a lazy map (except that imap accepts multiple iterators).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're planning on reorganizing a little to make a part-of-base Iterator or IterTools module, this would be a natural fit there under whichever final name we settle on.

immutable Generator{I,F}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any particular reason you chose to go with {I,F}, but ::F, ::I (for the order)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think Generator(func, range) is a bit more natural since it matches the order of comprehensions, but I suspect dispatching on the kind of iterator will be much more common than dispatching on the type of function. For example below I dispatch on Generator{IteratorND}.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generator(func, range) also allows Generator(range) do ... end syntax.

f::F
iter::I
end

start(g::Generator) = start(g.iter)
done(g::Generator, s) = done(g.iter, s)
function next(g::Generator, s)
v, s2 = next(g.iter, s)
g.f(v), s2
end

collect(g::Generator) = map(g.f, g.iter)
49 changes: 49 additions & 0 deletions base/iterator.jl
Original file line number Diff line number Diff line change
Expand Up @@ -292,3 +292,52 @@ eltype{I1,I2}(::Type{Prod{I1,I2}}) = tuple_type_cons(eltype(I1), eltype(I2))
x = prod_next(p, st)
((x[1][1],x[1][2]...), x[2])
end

_size(p::Prod2) = (length(p.a), length(p.b))
_size(p::Prod) = (length(p.a), _size(p.b)...)

"""
IteratorND(iter, dims)

Given an iterator `iter` and dimensions tuple `dims`, return an iterator that
yields the same values as `iter`, but with the specified multi-dimensional shape.
For example, this determines the shape of the array returned when `collect` is
applied to this iterator.
"""
immutable IteratorND{I,N}
iter::I
dims::NTuple{N,Int}

function (::Type{IteratorND}){I,N}(iter::I, shape::NTuple{N,Integer})
li = length(iter)
if li != prod(shape)
throw(DimensionMismatch("dimensions $shape must be consistent with iterator length $li"))
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ERROR: UndefVarError: a not defined

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks.

new{I,N}(iter, shape)
end
(::Type{IteratorND}){I<:AbstractProdIterator}(p::I) = IteratorND(p, _size(p))
end

start(i::IteratorND) = start(i.iter)
done(i::IteratorND, s) = done(i.iter, s)
next(i::IteratorND, s) = next(i.iter, s)

size(i::IteratorND) = i.dims
length(i::IteratorND) = prod(size(i))
ndims{I,N}(::IteratorND{I,N}) = N

eltype{I}(::IteratorND{I}) = eltype(I)

collect(i::IteratorND) = copy!(Array(eltype(i),size(i)), i)

function collect{I<:IteratorND}(g::Generator{I})
sz = size(g.iter)
if length(g.iter) == 0
return Array(Union{}, sz)
end
st = start(g)
first, st = next(g, st)
dest = Array(typeof(first), sz)
dest[1] = first
return map_to!(g.f, 2, st, dest, g.iter)
end
1 change: 1 addition & 0 deletions base/sysimg.jl
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ end
include("essentials.jl")
include("docs/bootstrap.jl")
include("base.jl")
include("generator.jl")
include("reflection.jl")
include("options.jl")

Expand Down
34 changes: 34 additions & 0 deletions doc/manual/arrays.rst
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,40 @@ that the result is of type ``Float64`` by writing::

Float64[ 0.25*x[i-1] + 0.5*x[i] + 0.25*x[i+1] for i=2:length(x)-1 ]

.. _man-generator-expressions:

Generator Expressions
---------------------

Comprehensions can also be written without the enclosing square brackets, producing
an object known as a generator. This object can be iterated to produce values on
demand, instead of allocating an array and storing them in advance
(see :ref:`_man-interfaces-iteration`).
For example, the following expression sums a series without allocating memory::

.. doctest::

julia> sum(1/n^2 for n=1:1000)
1.6439345666815615

When writing a generator expression with multiple dimensions inside an argument
list, parentheses are needed to separate the generator from subsequent arguments::

julia> map(tuple, 1/(i+j) for i=1:2, j=1:2, [1:4;])
ERROR: syntax: invalid iteration specification

All comma-separated expressions after ``for`` are interpreted as ranges. Adding
parentheses lets us add a third argument to ``map``::

.. doctest::

julia> map(tuple, (1/(i+j) for i=1:2, j=1:2), [1:4;])
4-element Array{Any,1}:
(0.5,1)
(0.333333,2)
(0.333333,3)
(0.25,4)

.. _man-array-indexing:

Indexing
Expand Down
42 changes: 28 additions & 14 deletions src/julia-parser.scm
Original file line number Diff line number Diff line change
Expand Up @@ -1401,21 +1401,22 @@
(parse-comma-separated s parse-eq*))

;; as above, but allows both "i=r" and "i in r"
(define (parse-iteration-spec s)
(let ((r (parse-eq* s)))
(cond ((and (pair? r) (eq? (car r) '=)) r)
((eq? r ':) r)
((and (length= r 4) (eq? (car r) 'comparison)
(or (eq? (caddr r) 'in) (eq? (caddr r) '∈)))
`(= ,(cadr r) ,(cadddr r)))
(else
(error "invalid iteration specification")))))

(define (parse-comma-separated-iters s)
(let loop ((ranges '()))
(let ((r (parse-eq* s)))
(let ((r (cond ((and (pair? r) (eq? (car r) '=))
r)
((eq? r ':)
r)
((and (length= r 4) (eq? (car r) 'comparison)
(or (eq? (caddr r) 'in) (eq? (caddr r) '∈)))
`(= ,(cadr r) ,(cadddr r)))
(else
(error "invalid iteration specification")))))
(case (peek-token s)
((#\,) (take-token s) (loop (cons r ranges)))
(else (reverse! (cons r ranges))))))))
(let ((r (parse-iteration-spec s)))
(case (peek-token s)
((#\,) (take-token s) (loop (cons r ranges)))
(else (reverse! (cons r ranges)))))))

(define (parse-space-separated-exprs s)
(with-space-sensitive
Expand Down Expand Up @@ -1471,6 +1472,9 @@
(loop (cons nxt lst)))
((eqv? c #\;) (loop (cons nxt lst)))
((eqv? c closer) (loop (cons nxt lst)))
((eq? c 'for)
(take-token s)
(loop (cons (parse-generator s nxt) lst)))
;; newline character isn't detectable here
#;((eqv? c #\newline)
(error "unexpected line break in argument list"))
Expand Down Expand Up @@ -1515,7 +1519,7 @@
(define (parse-comprehension s first closer)
(let ((r (parse-comma-separated-iters s)))
(if (not (eqv? (require-token s) closer))
(error (string "expected " closer))
(error (string "expected \"" closer "\""))
(take-token s))
`(comprehension ,first ,@r)))

Expand All @@ -1525,6 +1529,9 @@
`(dict_comprehension ,@(cdr c))
(error "invalid dict comprehension"))))

(define (parse-generator s first)
`(generator ,first ,@(parse-comma-separated-iters s)))

(define (parse-matrix s first closer gotnewline)
(define (fix head v) (cons head (reverse v)))
(define (update-outer v outer)
Expand Down Expand Up @@ -1953,6 +1960,13 @@
`(tuple ,ex)
;; value in parentheses (x)
ex))
((eq? t 'for)
(take-token s)
(let ((gen (parse-generator s ex)))
(if (eqv? (require-token s) #\) )
(take-token s)
(error "expected \")\""))
gen))
(else
;; tuple (x,) (x,y) (x...) etc.
(if (eqv? t #\, )
Expand Down
17 changes: 17 additions & 0 deletions src/julia-syntax.scm
Original file line number Diff line number Diff line change
Expand Up @@ -1926,6 +1926,23 @@
(lower-ccall name RT (cdr argtypes) args))))
e))

'generator
(lambda (e)
(let ((expr (cadr e))
(vars (map cadr (cddr e)))
(ranges (map caddr (cddr e))))
(let* ((argname (if (and (length= vars 1) (symbol? (car vars)))
(car vars)
(gensy)))
(splat (if (eq? argname (car vars))
'()
`((= (tuple ,@vars) ,argname)))))
(expand-forms
`(call (top Generator) (-> ,argname (block ,@splat ,expr))
,(if (length= ranges 1)
(car ranges)
`(call (top IteratorND) (call (top product) ,@ranges))))))))

'comprehension
(lambda (e)
(expand-forms (lower-comprehension #f (cadr e) (cddr e))))
Expand Down
26 changes: 26 additions & 0 deletions test/functional.jl
Original file line number Diff line number Diff line change
Expand Up @@ -178,3 +178,29 @@ let
foreach((args...)->push!(a,args), [2,4,6], [10,20,30])
@test a == [(2,10),(4,20),(6,30)]
end

# generators (#4470, #14848)

@test sum(i/2 for i=1:2) == 1.5
@test collect(2i for i=2:5) == [4,6,8,10]
@test collect((i+10j for i=1:2,j=3:4)) == [31 41; 32 42]
@test collect((i+10j for i=1:2,j=3:4,k=1:1)) == reshape([31 41; 32 42], (2,2,1))

let I = Base.IteratorND(1:27,(3,3,3))
@test collect(I) == reshape(1:27,(3,3,3))
@test size(I) == (3,3,3)
@test length(I) == 27
@test eltype(I) === Int
@test ndims(I) == 3
end

let A = collect(Base.Generator(x->2x, Real[1.5,2.5]))
@test A == [3,5]
@test isa(A,Vector{Float64})
end

let f(g) = (@test size(g.iter)==(2,3))
f(i+j for i=1:2, j=3:5)
end

@test_throws DimensionMismatch Base.IteratorND(1:2, (2,3))
2 changes: 2 additions & 0 deletions test/parse.jl
Original file line number Diff line number Diff line change
Expand Up @@ -347,3 +347,5 @@ end
# issue #14683
@test_throws ParseError parse("'\\A\"'")
@test parse("'\"'") == parse("'\\\"'") == '"' == "\""[1] == '\42'

@test_throws ParseError parse("f(2x for x=1:10, y")