Yes a VS project have a disk structure like:
Code:
...\Project
        Project.sln ...
        Debug
                Project.exe ...
        Release
                Project.exe ...
        Project
                stdafx.cpp / h
                file1.cpp
                project.vcxproj ...
                Debug
                        stdafx.obj ...
                Release
                        stdafx.obj ...
so those folders are normal.

A solution is kind of a container that holds all projects that are needed to build the final application. Let's say that you have some kickass library that you use in all your applications. You can of course link that library to the application by adding the path and so on to your application solution but what if you want to easily edit both your application source and the library source from the same IDE?

The solution() is to add the library project to your application solution and set the application project to depend on the library. Now you easily debug and fix both projects from the same IDE. Every time you build the application the library project also gets built if it has been changed. You don't have to remember to do that yourself.

So, when you delete a project from a solution you just delete the reference to it. The project files are left alone as they might be referenced by another solution.