local -> development server database issue

Hi,
Yesterday I upgraded from 5.1 to 5.3 on our local enviroment, all went not too badly and so I decided to merge the project with our development server which sits out on the WWW. It's on the same server as our main live site but connects to a different database (sitefinity_dev). The issue I had was I copied all the files up to our dev site, except the App_data/configuration folder and before I knew it my live site was giving me errors saying the database was more up to date that the site was!

How did this happen? The development site shouldn't touch the live database! Does sitefinity by default look for a database called sitefinity rather than using the DataConfig file that is already on the site, which in this case, pointed to sitefinity_dev database.

 

Hi Owain,

 Thanks for using Sitefinity.

You updated your local environment and copied the files (DLLS and such) to the dev machine without updating the sitefinity_dev database? This would cause the error. You will need to update the sitefinity_dev database, copy your whole upgraded project (including app_data/config) and change the dataconfig.config connectionstring to reflect the Sitefinity_Dev database again. Your update DLLs recognize the database is not update and that seems to be what error you're receiving.

If you have any questions please let me know.

Regards,
Patrick Dunn
the Telerik team

Hi Patrick,

I'm not sure if I have explained myself correctly.

I copied the site from local to Dev as the instructions say on your site to allow me to merge two sites to implement the upgrade. The way I understood is was if I merge the site, sitefinity should then automatically update my dev database. I didn't copy the config folder up to the dev location which I presumed would keep all my connection strings as they should be - pointing to my dev database.

But for some reason when I copied the files up to Dev, it then upgraded my Live database.

 

 

 

Dear Owain

This is a wild guess. Could it be that your local database by accident connected to the live site. If so you would have updated the live DB (by accident) then you upload all the bins and stuff but not the /configuration (which would normaly be correct) 

This would result in updated DB, updated .dlls but old config. That's the only logical explanation to me at the moment.

-----------------

Usually you DO NOT as you mentioned publish the .config files and once you uploaded the bins and access your site the first time the DB gets updated (and I assume the .configs changed) 

----------------

Check if you might by accident forgot to change the data.config SQL string in your dev envirement.

Kind regards 
Markus

In Sitefinity, the database connection string is located in App_Data > Sitefinity > Configuration > DataConfig.config.

The system version is in the same folder in system.config.

When Sitefinity starts up, it checks to see if the database is the same version as the source code. If not, it will run the update script on the database (the one in DataConfig.config)

Is it possible that your development environment had the public server's connection string by accident?

@Dan

I could not have written it better myself :-)

Markus

Learned the hard way. ;-)

I *strongly* recommend a copy only backup of your Sitefinity databases, and your deployment folders just before you attempt an upgrade.

@Dan

How do you think I came up with my reply? 

I have a staging and a live DB usually but both remote on the real server. I backup both databases daily so in case I accidentaly mess up one I have the other. 

But lets wait and see if Owain did the same as us.

Markus

Thanks all, it does sound like the system has had a look in my  system.config file, found that the connection strings don't match and then upgraded DB sitefinity.

To get round this issue, if I changed the connection string in system.config and Dataconfig.config both to my dev DB then merge the files - and not copy the config folder to DEV, would this solve the issue I had?

Just a bit nervous about upgrading again as really don't want my live site to go down again! :)

 

Off top of my head (I can double check my upgrade checklist when I get to my desk in 2 hours), the app_data > Sitefinity > configuration folder is one that always stays separate between dev and public. Never copy or upgrade that folder unless you have specific reason to do so. The upgrade process, when you are ready to do that on public server, will update the system.config file for you.

Always make a backup of the latest deployed folder and deployed database, and dev database (copy only backup) before upgrading. It will save you time to restore if you want to go back for some reason.

I learned about doing upgrades (after botching my first one in 4.0 -> 4.1) by setting it up a copy of my Sitefinity folder and database on a disconnected laptop. i ran through the process a few times, restoring from thumb drive. Then I compared the upgrade folder to thumb drive with a file compare tool to see what had been changed. Took a couple hours that first time, but it was worth it as upgrades are not as mysterious now. Worth doing if possible.

Hi Dan,

If you would be willing to share your upgrade document that would be really useful.

I did modify my dataconfig.config file so that I have 3 connection strings in it, one for local which is SQLExpress, one for dev which points to sitefinity_dev DB and one for live which points to sitefinity DB.

I had hoped that this would be bomb proof because :

Local can't see Dev or Live DB and can't connect to it via our network (internal or external due to firewall)

Comment out Local and Uncomment Dev -> Upload

Dev then wouldn't overwrite live because the names are different on the databases and then finally, uncomment live and comment out Local and Dev and the site is up and running.

Dev and Live are on the same server and point to the same datbase server but I can't be the only person that does this.

I have just upgraded local to 5.4 and would like to get DEV up to 5.4 this week but I'm sure you can all understand that I am a bit nervous about uploading the files to the server just incase it upgrades Live database again!

Question: Is it best just to upload the exact same folders that appear in the EmptyProject folder in Project Manager within my own project? Are these the only files that actually get upgraded?

 

Hi Owain,

 Thanks for Markus and Dan to getting back to you.

What they're saying is the the system config holds the version of Sitefinity.

for example. When Sitefinity loads it will check this version and then check the version in the database. If the version here is different than what's in the database it will execute the upgrade script against the database. When you performed the upgrade you likely still had the connectionString in your DataConfig.config file set to your production server. So, when you loaded up your upgraded (local) site it upgraded your production site instead of the one you had set up locally. This brought your site down because your Production site was using old DLLs, files, etc and a new database. Just be careful when you upgrade to set your connectionString to a local copy of the database. Once you have tested everything successfully locally you can then push your files to production. Kind regards,
Patrick Dunn
the Telerik team

Thanks again everyone for clearing this up for me. I'm going to get used to this as we have just bought another 3 licences for a 3 year term so will get plenty of practice!

I guess then when I download a site I shouldn't download the config folder either? i.e. if I am making a copy of production server to local so that I can start a new upgrade?

 

(Bummer... cannot upload my txt files. So I'll post my notes here. Sorry about the length.)

DISCLAIMER: This is NOT official documentation, use at your own risk.

How to Upgrade Sitefinity
=====================
 
NOTE: It is *VERY* important that you upgrade first on a development copy of the source code folder and recent copy of the database. Sitefinity upgrades may break the site and require some adjustments and a rebuild before it will work.

1. Use a duplicate copy of the source code folder and database. (SQL Server: Make a 'copy-only' backup of the live database and restore to a dev SQL server so you have the latest version of the data.) Make sure that the connection string in /App_Data/Sitefinity/Configuration/DataConfig.config points to this backup copy and NOT the live copy.

2. Review the Upgrade Notes for the version. Typically the steps are as follows:
 
3. Download & Install the latest Sitefinity "Project Manager" application.
 
4. Download the latest license file (*.lic) from the Sitefinity customer website. It

 is a version specific file.

5. Using the latest Sitefinity Project Manager, import (browse to) the copy of the source code you wish to upgrade. Click "Upgrade". It will update the .DLLs and possible make other changes.
 
6. Use Visual Studio to make the necessary changes to the project based on the Upgrade

 Notes. This generally includes web.config modifications and including new files into the project.

(Skip Step 7 if you do not use any third party Sitefinity plugins)

7. Since third party plugins do not always stay up to date, "Assembly Bindings" can be added to web.config to allow the Sitefinity DLLs to have different version numbers than the third party controls expect.

 Update the Assembly Bindings newVersion attribute to match the latest version of the Sitefinity DLLs in the bin folder.
 

 
8. Compile the project and resolve any remaining build errors.
 
9. Run your project. This will take a couple minutes longer than normal as the app spins up and then upgrades the schema of the development copy of the database. If this fails, worst case you may need to restore the database to a pre-upgrade copy before trying again. (step 1)
 
NOTE: This step is just an upgrade dry run. If you find any issues work through them before continuing.
 
10. Test the upgrade. Admin UI, public home page, Attachment files, any pages with custom widgets.
 
12. Follow the "How to Deploy" documentation to deploy this development upgrade to the live server. (I'll post it next)

(Owain: I believe you are looking for '6b' below. And "Rolling Back" step 3 if you have to roll back.)

DISCLAIMER: Not official documentation, use at your own risk.

How to Deploy a Sitefinity Update (The easy and reliable way)
=================================================
1. Develop and test your changes locally on a recent *copy* of the live database.   (See prior post)
 
2. Build to a local target folder.
 
3. Backup the /wwwroot folder on the live server. This is the last working source you are about to overwrite.
 
4. Copy Only Backup the most recent database. Preferred format is Application_Version_Date.bak.
   Example: Sitefinity_5_2_3800_nov20.bak
 
5. Take the IIS instance offline. (this may not be necessary)
 
6. Compare the local build against the deployed build on the server with a file compare tool such as WinMerge while considering the following:
 
6a. We have staff who modify CSS files on the live server. BAD idea, but works in a pinch, so they do it. Therefore, we must check to see if the CSS files are different than we had in our latest dev copy. Merge newer files with your build, test and check for conflicts.

6b. NEVER overwrite the live web server configuration files. These are located in:
/App_Data/Sitefinity/Configuration/*.config
These files are regularly updated with the Sitefinity Admin UI and likely have settings that are different than in development. On upgrades, these files are automatically generated.
 
Notable config files include DataConfig.config (datasource settings) and ToolboxesConfig.config (installed widgets)
 
When merging changes from your build to the live server, omit all files in this folder.
 
6c. The Web.config file may be different. You will likely need to compare the file using WinMerge to confirm that the differences are safe to move from the build copy to the live copy.
 
7. Restart the Sitefinity app pool for changes to take effect.
 
8. TEST your changes. Typically I test the home page, the Admin UI, and a couple pages that include custom widgets such as the HR Job Board, and open some attachments such as Word and PDF files to verify that they load correctly and do not produce a 404 error.

Common Issues (for us)
===================
Issue:    PDF Does not Download/Display Correctly after update.
Solution: You may have enabled file compression in web.config that is incompatible with the PDF format. RadCompression has been proven to be problematic. Do not enable it. If you enable any other IIS file compression, please test attachments to verify that they load. Typically Firefox and Chrome PDF viewers fail more often than IE.
 
Issue:    Widget does not display data
Solution: New Sitefinity version may have breaking changes. Verify that the widget code was not affected by an API change.
 
Verify that the Sitefinity API permissions for the widget show "Live" data (published) that can be viewed by anonymous readers.
 

Rolling Back
==========
1. Make sure the site is offline.
 
2. Restore your most recent Sitefinity backup
 
3. Replace the contents of the /App_Data/Sitefinity/Configuration folder with that from your most recent backup. Note that this folder MUST have read/write permissions for the IIS App Pool Account for the Sitefinity application. e.g. "IIS AppPool\Sitefinity". This allows the admin UI to make changes to the config files.
 
4. Replace the remaining wwwroot folder contents with that of your backup.
 
5. Restart the site.
 
6. Verify that the Sitefinity Administrative Site works, and test a number of other pages just as you would after an upgrade.

(Please post any errors here so they can be corrected.)

Fantastic! Thanks Dan!

 

I also pretty much follow what Dan does and found its a pretty safe process.

Some extra thoughts.

I use Web Deploy and as such use web.config transform allowing me to deploy web config differences between environments. (I host in Azure so need to change the a setting for Pod releases). If you use a SQL database, (Not SQL Azure) connection you can actually put that in the web config instead of the dataconfig file. (It will take precedence over the dataconfig... Well at least it did several versions ago)So you could manage your database connections safely there.

If you are using source control, (I use TFS in the cloud), make sure you Check out the whole SF project. Otherwise the project manager can run into problems with locked files and you end up with a half upgraded site with no idea what actually completed and what didn't.

I also place also the bin file files in a separate location and reference them, ensuring that copy local is true for each reference. I don't include in the project the bin folder. After the project manager upgrade I delete all the old SF references and then reference all the new sf dll's. By doing this I have avoid any dll mistakes.

 

Thanks Darrin,

You have brought something else up that I was hoping to ask the forum about and that is TFS. We have recently installed TFS and been struggling to get to grips with it - can you recommend any websites that show you how to use even just the basic check in / out process?

There are only two of us on the development team but we wanted to insure we didnt overwrite each other and we both work from one instance rather than 2 local copies of the site.

If anyone has any advice on using TFS and Sitefinity I would be interested in hearing it.

 

TFS 2010 or TFS 2012? I presume you are using a local version?

Have you ever had experience with a source control product before?

In general apart from what I mentioned above using Sitefinity in TFS, It is just like any other project. So any tutorials you see will be applicable. Try pluralsight. Very good Tuts there for TFS 2010, (They probably have ones for 2012 as well now.)

In regard to your main concern. When you check into TFS, (Or any source control system) If your changes do not fit safely with check in by your buddies then a merge conflict is raised and you get to resolve it.

One good practice should be that at the start of your day you do a Get Latest. This ensures any changes checked in by your buddies you get and are merged into your local workspace, (That is a folder on your PC), so you are always developing on the most current. Or have a system that when someone checks in everyone knows, so they know to get the changes at the next convenient moment.

Only check in completed work. If you want to save your stuff but its not finished and your afraid your PC harddrive may fail overnight, Use a shelf set and check that in. It acts like a special place to put your changes so they are safe.

TFS is a big topic, But I have no doubt you will learn it and refuse not to work with it after time. We should always, (almost always), use Soucre Control of some sort for our work. So good on you for taking the step.

We are using TFS 2012 and just upgraded our versions of VS to VS 2012. We have used source control back when we just did HTML sites with Dreamweaver but now looking at something a bit more robust with Visual Studio and Sitefinity. Will do some searching around the web to find tutorials on how to set it up correctly. I have something set up but it doesn't seem to be checking in and out correctly i.e. if I have something checked out, the other developer can also check that same file out and he gets no warning that I have it out.

Thanks for your reply,

So that is normal. (Not being notified) You can see in "Team Explorer" if items are checked out by others. But most of the time you don't need to care. What matters is when you check in and merge. Have a test between the both of you. Edit the same line in the same file and then check it in. See what happens. (If one of you don't get a merge conflict then something is wrong)

Still others like different setups so I am not saying you should work way. Check out the tuts and start with understanding the source control, check ins and checkouts. Then branching, then builds, then the Agile stuff. There is no need to know it all on day one. Good luck

We also moved from Dreamweaver to VS 2012 and TFS 2012. (back at SF 4.0 though)

The /bin folder does not do well under source control, and /app_data has some potential issues to watch out for. Otherwise, your project should be fine in TFS.

Some others on this forum may know better than I on this. But with the /bin folder it is probably best to only include the .dlls that came with Sitefinity. You will need to check them out when you upgrade. (and check-in any new .dlls). However, many other files in /bin are recompiled with your project as you work on it, so they will always be different than last check-in. They are not really needed under source control anyway.

As for /app_data as previously discussed, it is mostly temp files and config files that are specific to the setup (production or development). Putting them under source control will likely cause more problems than good. It's good to have a backup of that folder somewhere, but it will be managed more by the web interface than Visual Studio, and therefore a poor candidate for TFS.