Questions about the new development workflow

[Matteo]

Hi.
I’m stepping into the new development workflow and I’m trying to match some of the tasks we actually do with the behaviour of the new development workflow.

I’m starting from this post Missing environment color in the new ui where @louis activated the new development workflow on our project.

Our setup is the following:

• 1 “prod” environment - mapped to forest “production” environment type - visible to all the forest end users
• 1 “stg” environment - mapped to forest “staging” environment type - visible to all the forest end users (for UAT, training, tests, etc.)
• 1 “dev” environment - mapped to forest “development” environment type - not visible to the forest end users (to be used only by admins and developers)
  All the above environments have a server backend.

Then for each development feature branch, we used to have a forest development environment linked to the local workstation of the developer.

When deploying to dev/stg/prod, all the changes to the forest layout were repeated manually when needed. Obviously, the dev and stg environment required few attention then the prod one.

Now, with the new development workflow, I have some concerns.

What I understood from the docs is that as soon as I “start the migration process” towards the new development workflow, my “dev” environment will be removed. I’m not happy about this, because in the “dev” server and in the “dev” layout we have a stock of new features we have been developing for a long time and we are going to loose all the new layouts… because we’re not yet ready to go to prod with these new features.

By the way, it’s not production… let’s say that this is very annoying but not blocking actually.

Here are my concerns:

(1)

how can I have a “dev” environment mapped to a forest staging environment type, without showing this env to the end users?

(2)

Can we have two environments “dev” and “stg” both mapped to forest staging environment type?

(3)

What if we have a forest branch “A” and a forest branch “B”. Then we do the following:

• forest push “A” to the “dev” environment (which is a staging forest environment type) and
• forest push “B” to the “stg” environment (which is a staging forest environment type)

In this case, as far as I could understand, both “A” and “B” branches get removed. Because the only thing I can do is promote the changes to the prod environment with a forest deploy.
Let’s say I promote the changes we have in the “stg” enviroment (which is a staging forest environment type).

1. are these changes reflected to the “dev” environment (which is a staging forest environment type)?
2. in this scenario, is it then safe to forest deploy also the changes from the “dev” environment (which is a staging forest environment type)?

(4)

1. Is there any kind of “hierarchy” between my “dev” and “stg” enviroments (which are both mapped to staging forest environment type)? In other words, what I manually do in the “stg” layout gets reflected to “dev”? If no, how can I move back my layout changes from “stg” to “dev”? And what if in “dev” I have something pushed from a different forest branch?
2. how can I join my changes from “A” into “B” (“A” and “b” were described in (3) above), i.e. from “dev” to “stg” in order to join the changes before deploying both to “prod”?
3. seen that in my development CI/CD i need to get through “stg” before moving to “prod”, how can I remove the “B” changes from “stg” in order to push into “stg” only the features of “A”, to enable my UAT in “stg” and then promote to “prod”?

I am not so confident actually because I’ve already seen this post: Reset environment layout changes (dev workflow)
and without the option to reset the path triggered by a forest push it looks like the new development workflow prevents complex and parallel “git like” standard development habits on my side.

May you please help me in understanding the full picture? Because I’m not sure I’m right… maybe I’m missing something important

Thank you in advance,
Matteo

[Sliman_Medini]

Hi @Matteo,

I will try to answer all your questions
Maybe reading the “to be short section” at the end will be enough for your understanding.

@Matteo

Then for each development feature branch, we used to have a forest development environment linked to the local workstation of the developer.

With DWO, for each feature branch, the dev can create one forest branch inside his unique development environment. He can switch the forest branch (and switch his git branch at the same time) to work on another feature.

@Matteo

When deploying to dev/stg/prod, all the changes to the forest layout were repeated manually when needed. Obviously, the dev and stg environment required few attention then the prod one.

With DWO, when a developer have a forest branch with some layout changes, he can either:

• deploy it to production (this will apply the changes on every environments because the production is the “reference”, the “origin” for building layouts)
• push it to a staging environment (this will move the changes to the staging only, they disapear from the dev)

@Matteo

(1)

How can I have a “dev” environment mapped to a forest staging environment type, without showing this env to the end users?

It should not be the case. The dev environment is on the developer’s laptop and staging is on a server. Can you explain how this should help you? There are maybe other ways to achieve this.

@Matteo

(2)

Can we have two environments “dev” and “stg” both mapped to forest staging environment type?

No because of same answer to (1). But you can have multiple stagings.

@Matteo

(3)

What if we have a forest branch “A” and a forest branch “B”. Then we do the following:

• forest push “A” to the “dev” environment (which is a staging forest environment type) and
• forest push “B” to the “stg” environment (which is a staging forest environment type)

In this case, as far as I could understand, both “A” and “B” branches get removed. Because the only thing I can do is promote the changes to the prod environment with a forest deploy .
Let’s say I promote the changes we have in the “stg” enviroment (which is a staging forest environment type).

1. are these changes reflected to the “dev” environment (which is a staging forest environment type)?
2. in this scenario, is it then safe to forest deploy also the changes from the “dev” environment (which is a staging forest environment type)?

Because of answers to (2) and (1), you can’t have a dev mapped to a staging.
This is a normal way to work:
Figure a project with a prod, 3 staging and 10 devs

• a developer can push one of its branches onto a staging (this will make the branches changes appears only on the staging)
• a developer can deploy one of its branches (this will apply the changes on production, and then all environments will see this changes because all environments are following the production.)
• a developer can deploy a staging environment (same effect than previous)

“push” means a move of changes from a dev to a staging.
“deploy” means applying changes on (the production and so on) all the system.

To say it in another way, all environments are build by taking firstly the production layout (it’s the reference) and then applying the environment changes.

@Matteo

(4)

1. Is there any kind of “hierarchy” between my “dev” and “stg” enviroments (which are both mapped to staging forest environment type)? In other words, what I manually do in the “stg” layout gets reflected to “dev”? If no, how can I move back my layout changes from “stg” to “dev”? And what if in “dev” I have something pushed from a different forest branch?
2. how can I join my changes from “A” into “B” (“A” and “b” were described in (3) above), i.e. from “dev” to “stg” in order to join the changes before deploying both to “prod”?
3. seen that in my development CI/CD i need to get through “stg” before moving to “prod”, how can I remove the “B” changes from “stg” in order to push into “stg” only the features of “A”, to enable my UAT in “stg” and then promote to “prod”?

1. Today, there is very small hierarchy, production first, and everyone else is flat behind. But this is one thing we are thinking to implement.
2. just push dev to staging, then deploy staging.
3. you can’t remove only some changes from a branch.

To be short:
In DWO, the production is “the source of truth”
Then all other environments are getting their layout from the production.
Then all other environments are adding their own changes before render.

“push” allows you to move changes from dev to a staging
deploy allow to apply the changes from one environment onto the production.
Since all environments are based on production, the deploy command affects all environments. A contrario, the push command affects only the source and the destination of the command.

Please not hesitate if some points are not clear.
Best regards

[Matteo]

Hi @Sliman_Medini
Thank you very much for your detailed explanation. I understand.

It looks like with the new DWO @louis enabled on our project, we need to change our internal development flow. I’m sure you had good reasons to decide this, but this introduces on your side assumptions about the way we usually do things on our side. I understand that in some circumstances the capability to release the changes in the forest layout is a powerful tool. But with your DWO this comes at a cost on my side

Matteo:
How can I have a “dev” environment mapped to a forest staging environment type, without showing this env to the end users?

@Sliman_Medini:
It should not be the case. The dev environment is on the developer’s laptop and staging is on a server. Can you explain how this should help you? There are maybe other ways to achieve this.

We need to have multiple pre-prod environments. In our practice, we have different stages in the development flow and release process.
For example, we have a “shared” QA environment (we call it “dev” internally), where a number of tests are performed before releasing the features to the following stage. Then we may want to perform E2E tests with all the third party integrations. It looks like with the new DWO we can’t do this without showing to our end users.
The DWO assumption is: server envirnment ↔️ available to end users
I’m not happy about this. This is not how we do things. But it looks like I have no choice. Isn’t it?

Matteo
3. seen that in my development CI/CD i need to get through “stg” before moving to “prod”, how can I remove the “B” changes from “stg” in order to push into “stg” only the features of “A”, to enable my UAT in “stg” and then promote to “prod”?

@Sliman_Medini
3. you can’t remove only some changes from a branch.

In our development flow, sometimes we acknowledge that the feature is not ready during UAT. This means we forest pushed to the staging environment. With the new DWO we can’t revert the changes. So if in the mean time (while fixing these problems… or while waiting for approvals… or while waiting for some third party to fix thier integrations with our software…) we may want to move forward with some different feature. But we can’t forest push in the staging environment, because we don’t want to mix things up. So we need a different staging environment to forest push the changes we worked on.
But at the moment, our environment have been always managed as “sequential”. Like a progress pipeline. But this is not possible any more, because we’re locked in by the DWO.
Isn’t it?
What if we copy the layout from production to staging, in order to reset the staging environment? If this works, the only remaining problem is that we loose all the changes, because the branch was remuved as soon as we pushed it. Isn’t it? Why don’t you just let me decide if I want a branch to be removed when I forest push it?

One more question: what happens in case of conflicts? Does the last forest push and forest deploy overwrite whatever was changed in the destination? git has the concept of ancestor, thus the conflicts are caught. I understood that this does not happen in the DWO. Isn’t it?

In general, let me say that the new DWO introduces assumptions about our internal practices. Sure, everything is possible… and everything can be changed on our side… but I could have done without changes to my server environments, my flows, my project organisation, the pipelines…

I don’t want to seem argumentative, but if you introduce a “git like” flow on Forest side as a requirement for the dev flows on my side, this needs be flexible enough to let us do our job in the way we want to.

A different solution could have been serialising all the forest layout config, in order to manage it in our code base. This enables whatever flow we may want in our side, by managing all the changes with the development tools we use to have (git in our case). I also remember this was a feature request I upvoted somewhere, maybe in the product board… I can’t remember, actually.
Hope my feedback is appreciated as a contribution to you decisions about the features you introduce. I always appreciate the improvements to Forest, but this time I’m afraid I have concerns with DWO.

Thank you in advance for your answers.
Matteo

[louis]

Hey @Matteo ,

I also noticed that your project is running on an old version of our Roles sytem.

With our new Roles System, you can easily restrain access to environments to specific roles/group of users.

Simply uncheck the environments on the Roles settings.

I have also activated this feature on your project, hoping that it will ease your development workflow process. You can simply follow this guide. You need ALL yours env to run on version 6.6+.

On another note, we are thinking of a new command forest -o, --origin option for the branch command. In order to create a branch (dev env) from staging or any remote environments.

Do you think that would help in your process?

As always, thanks again for sharing this precious feedback

[Matteo]

Hi @louis
Thank you very much for your hint about the “roles” feature solution for the various environment restrictions.
I’ll check this out because this looks like the solution I was looking for.

About the new command, I think this should do the trick.
The flow could be:

1. forest branch and start developing
2. forest push -e staging to a stg environment
3. test are not fine, a revert is needed
4. forest branch -o staging to preserve a copy of the wip
5. forest environments:reset -e staging to allow the usage of the staging env for different purposes while fixing
6. fix… fix… fix…
7. GOTO (2)

…ad finally (and hopefully ), forest deploy

So, as far as I can tell, the -o flag to the branch command is the missing essential keystone
How long do you think it will approximately take to have this new feature available?

Another solution would be allowing the devs to decide if a branch must be deleted after a push or not… in order to be able to reuse if necessary.

Thank you very much,
Matteo

[louis]

Hi @Matteo,

You are welcome! I noticed you’ve already did the switch to our new roles system. I hope you like it.

Regarding the -o flag branch command, I can’t provide an ETA unfortunately. But it’s definitely in our short/mid term roadmap to continue iterating on the improvement of the developer experience.

[Matteo]

Hi @louis
yeah we switched to the new role system this morning, and it looks cool, so far

I’m willing to receive news about the new -o flag soon, because in my opinion it’s a quite blocking issue for the adoption of the DWO.

Thank you,
Matteo

[Matteo]

Hi @louis @Sliman_Medini
we’ve got a problem. I think it’s due to the new DWO you enabled some days ago.

We just released to our staging envoironment a new version of our server. This new version has a new smart action. But we can’t deal with the new smart action.

This is what we see un the collection configuration:

But in the project settings I can’t see the new smart action! So I can’t enable it:

Not sure if this is due to the new DWO or the new role system…
How can we enable this smart action??
Thank you.

[Matteo]

@louis @Sliman_Medini
I apologise. I was wrong. I didn’t realize I had to choose the staging env. in the left environments list. I was thinking that the purpose of the list was only to switch the envs on/on. Instead I just understood it’s a roles switcher too.

Thank you
Matteo

[louis]

No worries, tx for the update!

[Matteo]

Hi @louis

@louis

Regarding the -o flag branch command, I can’t provide an ETA unfortunately. But it’s definitely in our short/mid term roadmap to continue iterating on the improvement of the developer experience.

Do you have any update on this?

Thank you,
Matteo

[louis]

Hi @Matteo, yes, we discussed it internally and hope to have something by the end of Q2 this year (end of June). I’ll keep you posted on our progress!