• 12 Posts
  • 111 Comments
Joined 1 year ago
cake
Cake day: June 27th, 2023

help-circle













  • I haven’t used them in Spark directly but here’s how they are used for computing sparse joins in a similar data processing framework:

    Let’s say you want to join some data “tables” A and B. When B has many more unique keys than are present in A, computing “A inner join B” would require lots of shuffling if B, including those extra keys.

    Knowing this, you can add a step before the join to compute a bloom filter of the keys in A, then apply the filter to B. Now the join from A to B-filtered only considers relevant keys from B, hopefully now with much less total computation than the original join.






  • Thanks! I’ve seen some of your great posts and LED swap mods on BLF. These are well-photographed, something which is more difficult than I realized once I tried taking some photos of lights, with phones being so good now at taking pics of people and landscapes.

    I do have a ts10 in the rotation (plus two extras from when Terry from Wurkkos said the anodized colors were being discontinued) and it is as good as everyone says, although I only like it in 5700k.