Array operators

  • map keep all members,change members
  • filter keep some members,unchanged
  • map and filter combined (2 passes if not lazy implemented) keep some members,change members
  • reduce Reduce is more general,we can do the above and more. We can also carry any argument with us as state.
    map and filter can only see the current member

When we do array operations we need to use a fast function,because it is expected to run many times.

As being said on Collections intro MQL doesn't provide a fast operator to add at the end of an array.

This means that we can't reduce an array to an array in a fast way.
(conj uses $concat and if we make an array with 500+ members its very slow)

Alternatives

  • map and filter combination if possible (for cases when we want less members and changed members)
  • use javascript(its slow,but better than cMQL conj)
  • use cMQL reduce-array operator
    cMQL provides a fast reduce solution using $look-up(join with 1 dummy collection of 1 document), $facet,and accumulators.

cMQL uses this solution to group fast,but it can be used for arrays also.
Inside its group-by nil,and we can only use the few accumulators operators that we have.
Because of its limitations doesn't really solve the problem.

Examples

(drop-collection :testdb.testcoll)
(drop-collection :testdb.cmql)
(insert :testdb.cmql {}) ;;dummy 1 document collection in same database that group-array uses
(insert :testdb.testcoll [{:myarray [1 2 2 3]}
{:myarray [3 4 4]}])

Reduce (Array->single value)

(c-print-all
(q :testdb.testcoll
{:sum (reduce (fn [:a. :n.]
(+ :a. :n.))
0
:myarray)}))

Simple and very fast (as long as the function we use is fast)

Array->Array(map/filter)

(c-print-all
(q :testdb.testcoll
{:filtered (filter (fn [:m.] (> :m. 1)) :myarray)}))
;; map and keep all ($$REMOVE doesnt work to map,it adds nil)
(c-print-all
(q :testdb.testcoll
{:maped (map (fn [:m.] (+ :m. 1)) :myarray)}))
(c-print-all
(q :testdb.testcoll
{:maped-filtered (filter (fn [:m.]
(> :m. 1))
(map (fn [:m.] (+ :m. 1)) :myarray))}))

Simple to use and very fast (as long as the function we use is fast)
The last one the combination of filter and map,can be used as reduce alternative if we want to keep only some members and change them also.

Array->Array(reduce array operator)

(c-print-all
(q :testdb.testcoll
{:copy (reduce (fn [:a. :n.]
(conj :a. :n.))
[]
:myarray)}))

Simple but very slow if :myarray > 500 members,dont use in that case.

Array->Array (reduce cMQL operator)

Very fast no matter how big is the array,but pipeline stage not operator
=> results are added to the root document => i can later move it anywhere with assoc-in ,which is very fast

reduce is done using lookup with pipeline(with the dummy collection),facet,unwind,group its very fast and with combination of get-in and assoc-in of cMQL we can do it to any nested array,and then move it back into its place

(c-print-all
(q :testdb.testcoll
(reduce-array :myarray ;any expression that resolves to array,if nested i can use get-in etc
{:copy (conj-each :a)
:copyFilter (conj-each (if- (> :a 1)
:a
:REMOVE.))
:sum (sum :a)})))

I can use any MQL accumulator,here i use 3 accumulators.

Nested array

We can use reduce-array,for nested arrays also,and i can put the results back to the original locations.

Using cMQL's

  • get-in
  • assoc-in

The reduce-array is fast,the get-in is very fast,and the assoc-in is very fast.

(drop-collection :testdb.testcoll)
(insert :testdb.testcoll {:mymixedarray [1 {:a "b" :c {:d [1 2 3 2]}}]})
;; get-in,reduce,assoc-in to put it back in place
(c-print-all
(q :testdb.testcoll
(reduce-array (get-in :mymixedarray [1 "c" "d"]) ;;get the nested array
{:people (conj-each (if- (> :a 1)
:a
:REMOVE.))})
{:mymixedarray (assoc-in :mymixedarray [1 "c" "d"] :people)} ;;but it back
(unset :people)))