-
Notifications
You must be signed in to change notification settings - Fork 4
Collaboration #10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
spreeker
wants to merge
54
commits into
Attumm:nightly
Choose a base branch
from
spreeker:collaboration
base: nightly
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Collaboration #10
Changes from all commits
Commits
Show all changes
54 commits
Select commit
Hold shift + click to select a range
8335f11
new shiny create_model code that is more memory efficient
spreeker 69f6ade
enable mistaken disabled code
spreeker d7bdcab
update readme
spreeker a34870a
add column validation
spreeker 841bf49
reduce count was not registered fixed in template
spreeker aaa23b0
fix validation
spreeker a1bd933
start work on geo s2 stuff
spreeker 622f46e
wip s2 test
spreeker 9512c74
first working geo-selection
spreeker 613bdff
working geo-selection
spreeker 8ab6bb8
add curl test case for geo query
spreeker 93f801d
wip test with labels
spreeker 63d3429
geosearch works
spreeker c370cef
improve geojson handling
spreeker 1562973
show http table
spreeker 5059fef
keep lambda working for data without geometry
spreeker 2ec0762
create model now has ingore and geocolumn options
spreeker b5c3a2f
use gzipped csv
spreeker 0cc33c6
handle review remarks
spreeker 507f79b
add groupby / reduce to query and validate parameters
spreeker 7b83294
groupby cache experiment
spreeker 8a59246
start factoring bitarray into templateable code
spreeker 6b7be31
fix bugs using multiple bitarray keys
spreeker 0d224b9
fix cache headers
spreeker b4be4ae
update model template
spreeker 1fbfaf5
start with bitarray templateing
spreeker 197e3dc
bit array model code generation is working now
spreeker fda371f
add custom groupby
spreeker 0d525a5
add missing return
spreeker b893f60
validated and fixed reason for missing schools
spreeker 280b26a
add woning equivalent reduce
spreeker f6e924f
try pgzip
spreeker ba495a2
added bouwjaar
spreeker fff9bcb
added readlock, moved custom code
spreeker aac4b78
update readme
spreeker 041b486
fix bug returning raw item json
spreeker 1c9f9d1
allow reduce without groupby
spreeker 05a3627
add header column to csv
spreeker c2ff2d7
working merged build. labeledItems renamed to Items
spreeker d4a1ad6
remove merge mistake
spreeker ff690a7
remove merge mistake
spreeker 7faf26b
wip working new storage / retrieve methods
spreeker a7fc206
first working tests
spreeker e74c533
first geo testing wip
spreeker c87ba16
working geojson tests, removed some code duplication
spreeker 76c0d6b
add storage test, create example requests
spreeker cc247ce
wip: rewrite model creation, code, added column.go code
spreeker 4a306f6
wip: fix model creation after rewrite
spreeker e6f1e4f
done: code generation now works correctly, added new model and model_…
spreeker 5733650
fix: colom test
spreeker b51468a
docs: column.go
spreeker f8cb26e
docs: model.go creation
spreeker 29daf04
production version, improved error reporting about bitarray usage
spreeker bf9d65e
added huisnummer / toevoegingen many small fixes to code generation
spreeker File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
*.csv | ||
*.csv2 | ||
.git |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
extras/model.go | ||
testdata/* | ||
*.gz | ||
lambdadb |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,32 +1,78 @@ | ||
# LambdaDB | ||
In memory database that uses filters to get the data you need. | ||
Lambda DB has a tiny codebase which does a lot | ||
Lambda is not ment as a persistance storage or a replacement for a traditional | ||
Database but as fast analytics engine cache representation engine. | ||
|
||
Can be used for your needs by changing the models.go file to your needs. | ||
powers: https://dego.vng.nl | ||
|
||
## Properties: | ||
|
||
- Insanely fast API. 1ms respsonses | ||
- Fast to setup. | ||
- Easy to deploy. | ||
- Easy to customize. | ||
- Easy export data | ||
|
||
- Implement custom authorized filters. | ||
|
||
## Indexes | ||
|
||
- S2 geoindex for fast point lookup | ||
- Bitarrays | ||
- Mapping | ||
|
||
- Your own special needs indexes! | ||
|
||
## Flow: | ||
|
||
Generate a model and load your data. | ||
The API is generated from your model. | ||
Deploy. | ||
|
||
Condition: Your dataset must fit in memory. | ||
|
||
Can be used for your needs by changing the `models.go` file to your needs. | ||
Creating and registering of the functionality that is needed. | ||
|
||
|
||
### Steps | ||
You can start the database with only a csv. | ||
Go over steps below, And see the result in your browser. | ||
|
||
1. place csv file, in dir extras. | ||
2. `python3 create_model.py > ../model.go` | ||
3. cd ../ | ||
4. go fmt | ||
2. `python3 create_model_.py` answer the questions. | ||
3. go fmt model.go | ||
4. mv model.go ../ | ||
5. go build | ||
6. ./lambda --help | ||
7. ./lambda --csv assets/items.csv or `python3 ingestion.py -b 1000` | ||
9. curl 127.0.0.1:8128/help/ | ||
10. browser 127.0.0.1:8128/ | ||
|
||
|
||
11. instructions curl 127.0.0.1:8128/help/ | python -m json.tool | ||
|
||
|
||
|
||
### Running | ||
|
||
sudo docker-compose up --no-deps --build | ||
|
||
promql {instance="lambdadb:8000"} | ||
|
||
python3 extras/ingestion.py -f movies_subset.tsv -format tsv -dbhost 127.0.0.1:8000 | ||
======= | ||
|
||
1. instructions curl 127.0.0.1:8000/help/ | python -m json.tool | ||
|
||
### Questions | ||
|
||
|
||
|
||
### TODO | ||
|
||
- load data directly from a database (periodic) | ||
- document the `create_model.py` questions | ||
- use a remote data source | ||
- use some more efficient storage method (done) | ||
- generate swagger API | ||
- Add more tests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,148 @@ | ||
package main | ||
|
||
import ( | ||
"errors" | ||
"fmt" | ||
"github.com/Workiva/go-datastructures/bitarray" | ||
"log" | ||
"strings" | ||
) | ||
|
||
type fieldIdxMap map[string]uint32 | ||
type IdxFieldMap map[uint32]string | ||
|
||
/* | ||
MappedColumn store fields in Idx | ||
*/ | ||
|
||
type MappedColumn struct { | ||
Idx fieldIdxMap // stores field to int values | ||
Field IdxFieldMap // stores int to field values to recover actual fields | ||
IdxTracker uint32 | ||
Name string | ||
} | ||
|
||
type ColumnRegister map[string]MappedColumn | ||
|
||
var RegisteredColumns ColumnRegister | ||
|
||
func init() { | ||
RegisteredColumns = make(ColumnRegister) | ||
} | ||
|
||
func NewReapeatedColumn(column string) MappedColumn { | ||
m := MappedColumn{ | ||
make(fieldIdxMap), | ||
make(IdxFieldMap), | ||
0, | ||
column, | ||
} | ||
RegisteredColumns[column] = m | ||
return m | ||
} | ||
|
||
// Store field name as idx value and idx as field value | ||
func (m *MappedColumn) Store(field string) { | ||
|
||
if _, ok := m.Idx[field]; !ok { | ||
m.Idx[field] = m.IdxTracker | ||
m.Field[m.IdxTracker] = field | ||
m.IdxTracker++ | ||
} | ||
} | ||
|
||
// Store Array field (postgres Array). | ||
func (m *MappedColumn) StoreArray(field string) []uint32 { | ||
|
||
fieldsArray := make([]uint32, 0) | ||
|
||
// parsing {a, b} array values | ||
// string should be at least 2 example "{}" == size 2 | ||
if len(field) > 2 { | ||
fields, err := ParsePGArray(field) | ||
|
||
if err != nil { | ||
log.Fatal(err, "error parsing array ") | ||
} | ||
|
||
for _, gd := range fields { | ||
m.Store(gd) | ||
} | ||
|
||
for _, v := range fields { | ||
fieldsArray = append(fieldsArray, Gebruiksdoelen.GetIndex(v)) | ||
} | ||
} | ||
return fieldsArray | ||
} | ||
|
||
func (m *MappedColumn) GetValue(idx uint32) string { | ||
return m.Field[idx] | ||
} | ||
|
||
func (m *MappedColumn) GetArrayValue(idxs []uint32) string { | ||
|
||
result := make([]string, 0) | ||
for _, v := range idxs { | ||
vs := m.GetValue(v) | ||
result = append(result, vs) | ||
} | ||
return strings.Join(result, ", ") | ||
} | ||
|
||
func (m *MappedColumn) GetIndex(s string) uint32 { | ||
return m.Idx[s] | ||
} | ||
|
||
// SetBitArray | ||
func SetBitArray(column string, i uint32, label int) { | ||
|
||
var ba bitarray.BitArray | ||
var ok bool | ||
|
||
// check if map of bitmaps is present for column | ||
var map_ba fieldBitarrayMap | ||
|
||
if _, ok = BitArrays[column]; !ok { | ||
map_ba := make(fieldBitarrayMap) | ||
BitArrays[column] = map_ba | ||
} | ||
|
||
map_ba = BitArrays[column] | ||
|
||
// check for existing bitarray for i value | ||
ba, ok = map_ba[i] | ||
|
||
if !ok { | ||
ba = bitarray.NewSparseBitArray() | ||
map_ba[i] = ba | ||
} | ||
// set bit for item label. | ||
ba.SetBit(uint64(label)) | ||
} | ||
|
||
func GetBitArray(column, value string) (bitarray.BitArray, error) { | ||
|
||
var ok bool | ||
|
||
if _, ok = BitArrays[column]; !ok { | ||
return nil, errors.New("no bitarray filter found for " + column) | ||
} | ||
|
||
// convert string value to actual indexed int. | ||
i, ok := RegisteredColumns[column].Idx[value] | ||
|
||
if !ok { | ||
msg := fmt.Sprintf("no indexed int value found for %s %s", column, value) | ||
return nil, errors.New(msg) | ||
} | ||
|
||
ba, ok := BitArrays[column][i] | ||
|
||
if !ok { | ||
msg := fmt.Sprintf("no bitarray found for %s %s %d", column, value, i) | ||
return nil, errors.New(msg) | ||
} | ||
|
||
return ba, nil | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great to to read!!!