Create a Ceph Cluster with Deepsea

How to setup a ceph cluster with Deepsea on top of SUSE Leap VMs.

  1. Install three VMs and configure "/etc/hosts" -> easiest way to do is to add all nodes in "/etc/hosts" on all VMs

  2. Install "salt-master" on one VM "zypper in salt-master"

  3. Install "salt-minion" on ALL VMs "zypper in salt-minion"

  4. Configure "/etc/salt/minion to point to the right master and restart minions

  5. Accept keys on the salt-master "salt-key -A -y"

  6. Clone the Deepsea repo "git clone deepsea" -> https://github.com/SUSE/DeepSea.git

  7. Edit "Deepsea/srv/pillar/ceph/master_minion.sls" -> to point to the salt master

  8. Inside the Deepsea folder run "make install"

  9. On the salt master run "systemctl restart salt-master"

  10. Run "chown -R salt /srv/pillar/*"

  11. Edit "/srv/modules/runners/validate.py" and comment "v.storage" and "v.ganesha" -> there could be more in the future

  12. Edit "/srv/modules/runners/populate.py" and change "if quantity > 3:" to "if quantity > 0:" -> otherwise you'll have a minimum of four VMs

  13. Run "salt-run state.orch ceph.stage.0" or "salt-run state.orch ceph.stage.prep"

  14. Run "salt-run state.orch ceph.stage.1" or "salt-run state.orch ceph.stage.discovery"

  15. Create "/srv/pillar/ceph/proposals/policy.cfg". Example:

    # Cluster assignment
    cluster-ceph/cluster/*.sls
    # Hardware Profile
    profile-*-1/cluster/*.sls
    profile-*-1/stack/default/ceph/minions/*yml
    # Common configuration
    config/stack/default/global.yml
    config/stack/default/ceph/cluster.yml
    # Role assignment
    role-master/cluster/*.sls
    role-mon/cluster/*.sls
    role-mon/stack/default/ceph/minions/*.yml
    
  16. Run "salt-run state.orch ceph.stage.2" or "salt-run state.orch ceph.stage.configure"

  17. Run "salt-run state.orch ceph.stage.3" or "salt-run state.orch ceph.stage.deploy"

  18. You should change the pg(p)_num to get rid of the "to few pgs per osd" error "ceph osd pool set rbd pg_num 120" and "ceph osd pool set rbd pgp_num 120"

You now should have an up and running cluster

Now you could add the openATTIC OBS (openSUSE Build Service) repo and install openATTIC and the openattic-deployment package

  1. Add the needed Repo from OBS "zypper ar http://download.opensuse.org/repositories/filesystems:/openATTIC/openSUSE_Leap_42.2/filesystems:openATTIC.repo"
  2. Install openATTIC "zypper in openattic openattic-module-ceph-deployment" -> you have to break dependencies because deepsea isn't shipped with leap by default
  3. Run "chmod 655 /etc/ceph/ceph.*"
  4. Run "openattic install"

You're done :)

We're hiring: Senior Backend Developer

Last year, we opened up a position for a senior frontend developer. This position has now been filled and we're very excited to welcome Ricardo Marques to our team! Ricardo just concluded his first week and will support our team from out of his home office near Lisbon, Portugal. He started contributing to openATTIC a while ago already, and we look forward to his future contributions.

But we're still hiring; we've just opened a new position for the position of a Senior Backend Developer Enterprise Storage Management.

In this role, you'll be working with the openATTIC team on adding new features to openATTIC's Python/Django-based backend code, as well as improving and extending existing functionality.

See the job opening for further details on our expectations and requirements. If you have any questions or would like to learn more, don't hesitate to get in touch with us!

By the way, if you're interested in working on Linux and Open Source Software in general, SUSE currently has 50+ job offerings available!

Using Docker for openATTIC Development

/images/docker-logo-small-h.png

In addition to using Vagrant for working on the openATTIC code base, it is now also possible to set up a development environment based on Docker containers.

Thanks to our colleague Ricardo Dias from the SUSE Enterprise Storage Team, you can spin up docker images based on either openSUSE Leap 42.2 or Ubuntu 16.04 LTS "Xenial" that include everything to get openATTIC up and running. The Docker container actually maps the Mercurial code repository from the host development system into the runtime environment.

This way, you can use your preferred editor of choice and any local changes can be tested in the container immediately.

You can find the required Dockerfiles and support scripts in the openattic-docker git repository on github. The README.md file included in the repository explains how to get started.

We hope, that this gives you more choice for setting up a developent environment for openATTIC quickly, without having to make any changes to your host operating system.

As usual, we welcome your feedback and contributions!

Moved this blog to git and BitBucket

This blog is built a static web site, powered and managed by the excellent Nikola static web site generator.

The motivation for using a static blog was to make writing articles more straightforward for our developers, without introducing unnecessary media breaks and barriers. Using this framework, one can contribute content using familiar tools and workflows.

When we initially created this blog in September 2015, we used Mercurial for managing the source files and assets and hosted the repository on an internal Mercurial host.

This has now been changed - we moved the source code repository to BitBucket, as part of the openATTIC umbrella project.

We also converted the repository from Mercurial to git along the way, primarily as an exercise to evaluate how easy such a conversation can be performed, but also as a first step to give us a way to practice our git skills and to further open up our development processes.

The conversion was performed on a local development system using the hg-fast-export Python script and was quite simple, following the steps outlined in the README.md file:

$ mkdir openatticblog.git
$ cd openatticblog.git
$ git init
$ ../fast-export/hg-fast-export.py -r ../openatticblog.hg

Afterwards, the git repo could easily be pushed to BitBucket.

BTW, if you take a closer look at this repo, you will notice there's a branch named website. This is a work in progress project to convert the existing openattic.org web site from Typo3 to Nikola.

While the content is shaping up nicely, the layout and design still needs some more attention before we will make the switch. If you're familiar with creating web sites using Nikola and Bootstrap and want to give us a hand, please let us know!

openATTIC 2.0.18 beta has been released

We are very happy to announce the first openATTIC release in 2017. Although it is not as extensive as usual it contains some significant highlights:

This release comes with a very new feature - the implementation of a Taskqueue module. Tasks that take more time, for example the creation of placement groups in a Ceph cluster, can now be controlled in the openATTIC GUI. The Taskqueue adds the functionality to create tasks, run them in the background, track their status and collect their results. We also added some sections about background tasks and the Taskqueue usage to our documentation.

Furthermore, we've made some usability improvements. The installed version of all known openATTIC hosts is displayed in the GUI now. Buttons and pages without any function have been removed from the user interface.

Read more…

Speaking about openATTIC at FOSDEM 2017

https://fosdem.org/2017/assets/style/logo-big-a5243e4d7e00f8bc6816e2b3f3804f505a17ae4832e6e52a24d183617e03a87c.png

On February 4th and 5th, the annual FOSDEM conference will take place in Brussels, Belgium.

This year, I'll give a talk titled Ceph and Storage management with openATTIC in which I'd like to give an overview and update about our project.

The talk will be a session in the the Software Defined Storage developer room that is scheduled to take place on Sunday, 5th. My talk takes place at 13:30.

FOSDEM is a very popular and intensive conference - I look forward to attending it again!

openATTIC 2.0.17 beta has been released

Shortly before the holidays, we're happy to announce the availability of openATTIC version 2.0.17!

Due to the onboarding of the openATTIC team to SUSE, this release took a bit longer than the usual cycle. But we hope it was worth waiting for!

As usual, we included a good mix of bug fixes, improvements and some new functionality. Some highlights in this version include:

  • Lots of improvements for installing and running openATTIC on Ubuntu Linux 16.04 aka "Xenial Xerus". openATTIC on Xenial now passes all tests and the installation should be fairly straightforward. We're still interested in your feedback, though - please let us know if you still find any issues installing and running openATTIC on this platform. An installation how-to is available in the last Release announcement, an update of the installation documentation will also be provided.
  • In the Ceph backend, we moved calls to librados into separate processes, to prevent potential blocking of the Django application and Web UI in case that RADOS calls get stuck.
  • We've replaced the previous systemd DBUS calls with calls to systemctl and now use systemd for starting/stopping/reloading services on all platforms where systemd is available (previously, openATTIC was still using the "old" SysV init tools, e.g. service). We also switched to using reload-or-restart for reloading services by default.
  • The Web UI received a number of refinements and improvements, e.g. some API-Recorder fixes and usability enhancements. Now, it's also possible to obtain a user's authentication token via the web interface, which helps to avoid using passwords in external scripts or applications that want to access the openATTIC REST API.
  • During the initial installation on RPM-based systems, the openattic PostgreSQL database and user account are now created using a random password via oaconfig install.
  • Improvements for setting up development environments using Vagrant.
  • We also added a new chapter to the Documentation that describes how to set up a multi-node configuration.

Read more…

openATTIC Wins a Silver OSBAR Award 2016

/images/osbar-logo-small.png

Since 2014, OSBAR, the innovation award of the German Open Source Business Alliance (OSB Alliance), highlights open source projects that add real benefit to the IT-world.

Submissions are assessed based on originality, innovation, practical relevance and maturity by a committee of six well-known German IT- and open source experts.

In total, 20 open source projects applied for this year's awards and one of them was our open source Ceph and storage management framework.

Read more…

Status of openATTIC on Ubuntu Xenial

We have been working on porting openATTIC to Ubuntu 16.04 LTS "Xenial Xerus" for quite some time now, and we wanted to give you a quick update on the current status as of openATTIC version 2.0.16.

It turns out that Xenial provides a number of challenges and differences that we needed to take into account, for example a new version of Django and the Django REST framework, as well as some additional underlying changes.

Making all the required changes in a backwards-compatible manner and testing them is quite time-intensive.

In a nutshell, we're not quite there yet, but we're making progress.

Some of these issues can be worked around, but the overall "out of the box experience" still needs to be further improved.

Read more…

We're hiring: Senior Frontend Developer

Now that we've joined SUSE, we're able to extend the team working on openATTIC!

We've just opened a new position and are now hiring for the position of a "Senior Frontend Developer Enterprise Storage Management".

In this role, you'll be working with the openATTIC team on adding new features to openATTIC's web-based management frontend, as well as improving and extending existing functionality.

The openATTIC web interface is based on well-known web development technologies like AngularJS and Bootstrap and communicates with the openATTIC backend via it's REST API.

See the job opening for further details on our expectations and requirements. If you have any questions or would like to learn more, don't hesitate to get in touch with us!

By the way, if you're interested in working on open source software, SUSE currently has 70+ job offerings available!

Update 2017-02-17: this position has been filled in the meanwhile (welcome Ricardo Marques to our team!)

However we're now hiring for a senior backend developer role!