Key/Value backend #865

mogorman · 2018-02-04T22:58:15Z

This is just the same as #624 I will make the other hundred refrence changes needed but i just wanted to get this started.

There are quite some ideas in the issue tracker about how we could improve nixops radically.

I would like to start out with a feasible iterative plan. As far as I can see, only very trivial SQL operations are every executed against the sqlite3 database; the main use case is to use transactions to support rollback in the case of exceptions. This should not be to hard to implement ourselves in memory, in combination with a key/value server that supports locking certain keys.

Needed to faclitate multiple backends (like sqlite3/other remote database)

… it in state file implementation to __db

For now it's just the underlying DB, but it resembles something that can roll back the current transaction.

mogorman · 2018-02-14T20:32:13Z

I reduced the scope of this back to just the sqlite and json work. I will make another fork to add in mysql/postgresql support. What is here works. I also added a unit test for testing the json backend, but couldnt figure out how to run just it i tried

python2 tests.py tests.functional.generic_json_deployment_test

but it kept giving me libvirt errors. I didn't have interest in running all the tests on my laptop. is there a way to kick off the bot on it @grahamc

I am done with this iteration unless others have requests I think it is good to go.

moretea · 2018-02-17T07:51:43Z

@rbvermaas I also ran into those libvirt issues. I remember that you said that those have not worker properly at all?

@mogorman could you check if those tests do, or do not work on the master branch?

mogorman · 2018-02-17T13:01:32Z

@moretea if im not mistaken ive fixed all the tests but i was hoping someone with gce, aws, etc could run them all

moretea · 2018-02-17T15:09:56Z

As far as I can remember, I tried the local virtualbox backend, and GCE. Both worked.

I could check out AWS & GCE later, but am on a holiday this weekend and don't have time 🙈

AmineChikhaoui · 2018-02-18T20:09:22Z

Hi @mogorman , VPC tests seem to be broken e.g:

Traceback (most recent call last):
  File "/nix/store/f8phnzpvr67l1r18xbhbwsr4cskqacys-python-2.7.14/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/f8phnzpvr67l1r18xbhbwsr4cskqacys-python-2.7.14/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/amine/src/nixops/tests/__init__.py", line 18, in destroy
    depl = sf.open_deployment(uuid)
  File "/home/amine/src/nixops/nixops/state/sqlite_connector.py", line 148, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/amine/src/nixops/nixops/state/sqlite_connector.py", line 144, in _find_deployment
    return nixops.deployment.Deployment(self, res[0][0], sys.stderr)
  File "/home/amine/src/nixops/nixops/deployment.py", line 78, in __init__
    self.resources = self._state.get_resources_for(self)
  File "/home/amine/src/nixops/nixops/state/sqlite_connector.py", line 242, in get_resources_for
    r = self._create_state(deployment, type, name, id)
  File "/home/amine/src/nixops/nixops/state/sqlite_connector.py", line 394, in _create_state
    return cls(depl, name, id)
  File "/home/amine/src/nixops/nixops/resources/vpc.py", line 43, in __init__
    self.vpc_id = self._state.get('vpcId', None)
  File "/nix/store/f8phnzpvr67l1r18xbhbwsr4cskqacys-python-2.7.14/lib/python2.7/_abcoll.py", line 382, in get
    return self[key]
  File "/home/amine/src/nixops/nixops/state/state_helper.py", line 22, in __getitem__
    value = self._state.get_resource_attr(self.uuid, self.id, name)
NameError: global name 'name' is not defined

tomberek · 2018-02-18T21:16:30Z

I like using nixops-manager and ran into this issue. Seems like it has to do with escaping in JSON.

[nix-shell:~/cloud]$ ./manage gce create '<cloud.nix>'                         
created deployment ‘56c8f9ee-14ef-11e8-aad6-00163e000000’                      
56c8f9ee-14ef-11e8-aad6-00163e000000 

[nix-shell:~/cloud]$ ./manage gce deploy                                                                                                                      
error: getting status of ‘/home/dev/cloud/\<cloud.nix\>’: No such file or directory                                                                           
error: evaluation of the deployment specification failed    

[nix-shell:~/cloud]$ cat gce.json      
{
  "schemaVersion": 0, 
  "deployments": {
    "56c8f9ee-14ef-11e8-aad6-00163e000000": {
      "attributes": {
        "nixExprs": "[\"/home/dev/cloud/\\\\<cloud.nix\\\\>\"]", 
        "name": "gce"
      }, 
      "resources": {}
    }
  }
}

using a normal (non bracketed path) seems to work well for gce

[nix-shell:~/cloud]$ ./manage gce create './cloud.nix'                                                                                                        
created deployment ‘d3fcdf3d-14ef-11e8-900d-00163e000000’                      
d3fcdf3d-14ef-11e8-900d-00163e000000   

[nix-shell:~/cloud]$ ./manage gce deploy                                       
bootstrap> creating GCE image 'n-d3fcdf3d14ef11e8900d00163e000000-bootstrap'... 

...
[nix-shell:~/cloud]$ ./manage gce destroy
warning: are you sure you want to destroy GCE machine 'n-d3fcdf3d14ef11e8900d00163e000000-single'? (y/N) y
single...> destroying the GCE machine...
...

mogorman · 2018-02-18T21:58:28Z

Thanks I'll look at these now

backends

mogorman · 2018-02-19T07:11:47Z

@AmineChikhaoui I believe i fixed these unit test failures with my last round of commits, they fail for me stil as I don't have aws keys but they do correctly update the attrs and get past the failure point you posted.

@tomberek Your issue is coming from nix-shell over escaping things. probably because you were in the dev-shell and then executed another deeper nix-shell. can you try installing this fork of nixops and running without the nested shells? this error does not happen to me when i used <> directly

{
  "schemaVersion": 0, 
  "deployments": {
    "8f9506b0-153e-11e8-839e-024290535936": {
      "attributes": {
        "nixExprs": "[\"<trivial.nix>\", \"<trivial-vbox.nix>\"]", 
        "name": "test", 
        "configsPath": "/nix/store/dm9a1cixp8w7mylzv6vhpwv7nj0i2kqr-nixops-machines", 
        "description": "Web server"
      }, 
      "resources": {}
    }
  }
}

AmineChikhaoui · 2018-02-19T09:46:26Z

@mogorman Thanks, so the previous errors are gone but I still see some errors during the deployment tear down in the unit tests, can't post them now because it exposes some resources IDs and AWS debug output but I'll re-run it later this week and look at the errors.

mogorman · 2018-02-23T02:46:43Z

@AmineChikhaoui did you ever get a chance to get me more info as to what is broken? or can anyone help me understand how to run the test framework better? the only errors i get when running tests are those caused by lack of aws gce etc credentials so i assume the other tests passed?

andrewchambers · 2018-03-09T05:17:07Z

Could someone explain the rationale for this change? what is the point?

mogorman · 2018-03-09T05:59:09Z

the point of providing a plugable state backend is to make it easier for people to work with nixops. In the case of the json one, it makes it much easier to commit these things to git and share them among a team, in the future I want to make a mysql/postgresql backend as well so that I can manage the state there. Others have requested a backend using s3 and simpledb. this work makes it easier to do all these things.

moretea · 2018-03-09T17:11:02Z

Having a json file is the starting point to prove that we can use any K/V store. Once that's done, adding e.g. mysql, or S3 + DynamoDB should just work fine.

gleber-da · 2018-04-26T12:13:38Z

👍 to these changes. These would help us share the workload of managing our deployments in the team.

nh2 · 2018-05-27T22:08:58Z

What's the state currently? Do you only need testers for the 3 remaining backends?

moretea · 2018-05-29T06:42:35Z

@nh2 talking to me? And which backends are you referring to?

nh2 · 2018-05-29T11:35:19Z

To whomever would like this PR in, really. I was referring to

None backend done, needs to be verified

GCE done, needs to be verified

AWS done, needs to be verified

Hetzner done, needs to be verified

in the issue description.

tomberek · 2018-11-04T19:24:34Z

What is needed for this? Do you need user reports and testing?

domenkozar · 2020-03-26T17:33:53Z

See #1264

grahamc · 2020-03-26T19:22:16Z

Hello!

Thank you for this PR.

In the past several months, some major changes have taken place in
NixOps:

Backends have been removed, preferring a plugin-based architecture.
Here are some of them:
NixOps Core has been updated to be Python 3 only, and at the
same time, MyPy type hints have been added and are now strictly
required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
release. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work
on.

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.

Graham

moretea added 30 commits March 13, 2017 19:42

Move statefile into state.file module

7035abe

Introduce different state state schemes.

6da031f

Needed to faclitate multiple backends (like sqlite3/other remote database)

Rename statefile->state in deployment.py

8269650

Move file based deployment lock to file state implementation.

c73773f

Fix indenting

8a4e34a

Bite the bullet; remove _db references from business objects + rename…

bb056ea

… it in state file implementation to __db

move get_resources_for

6cc171a

move set_deployment_attrs

813a231

move del_deployment_attr

7b58934

move get_deployment_attr

7ed3fd3

move create_resource

0db686d

Move most logic in export() to get_all_deployment_attrs in statefile.

97e0714

Make import use an atomic transaction thingy.

584d658

For now it's just the underlying DB, but it resembles something that can roll back the current transaction.

move clone_deployment

97bda1a

move part of delete_resource

77e9b93

missed a few _'s

4d33f93

move part of delete to _delete_deployment

5649e39

use state atomic

4075901

Move part of logic of rename to _rename_resource

5f39278

oh yeah, I did rename _db to __db!

6605128

Use atomic instead of _db

5881c06

Use atomic instead of _db

de9d700

move set_resource_attrs

ff23c1c

move del_resource_attr

eb2a24a

move get_resource_attr

47b3a43

move part of export to get_all_resource_attrs

6835823

Use atomic instead of _db

3739051

oh yeah, I did rename _db to __db!

66dee1d

simple typo's

6254ccb

Need to pass in object; too much coupling to refactor right now

2d1a822

mogorman added 4 commits February 14, 2018 14:17

remove debugging

76ffca7

adding unit test and testing it

6cf8c02

new unit tests

f656d58

remove debugging

83c4af9

mogorman added 3 commits February 18, 2018 21:43

fix variable name issue and move subclasses to common spot among

4cea4d2

backends

work on unit tests

f477f2a

make json match the behavior of sqlite

36f2479

misnamed variable

0a836b6

edolstra mentioned this pull request Nov 2, 2018

SQLite should not be a hard dependency #1036

Closed

grahamc closed this Mar 26, 2020

Uh oh!

Key/Value backend #865

Key/Value backend #865

Uh oh!

Conversation

mogorman commented Feb 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mogorman commented Feb 14, 2018

Uh oh!

moretea commented Feb 17, 2018

Uh oh!

mogorman commented Feb 17, 2018

Uh oh!

moretea commented Feb 17, 2018

Uh oh!

AmineChikhaoui commented Feb 18, 2018

Uh oh!

tomberek commented Feb 18, 2018

Uh oh!

mogorman commented Feb 18, 2018

Uh oh!

mogorman commented Feb 19, 2018

Uh oh!

AmineChikhaoui commented Feb 19, 2018

Uh oh!

mogorman commented Feb 23, 2018

Uh oh!

andrewchambers commented Mar 9, 2018

Uh oh!

mogorman commented Mar 9, 2018

Uh oh!

moretea commented Mar 9, 2018

Uh oh!

gleber-da commented Apr 26, 2018

Uh oh!

nh2 commented May 27, 2018

Uh oh!

moretea commented May 29, 2018

Uh oh!

nh2 commented May 29, 2018

Uh oh!

tomberek commented Nov 4, 2018

Uh oh!

domenkozar commented Mar 26, 2020

Uh oh!

grahamc commented Mar 26, 2020

Uh oh!

Uh oh!

mogorman commented Feb 4, 2018 •

edited

Loading