If you installed MSDE from the RMS CD, you were asked for a password during the install, and whatever you entered there is the sa password. If you already had MSDE or SQL Server installed, the password may have been blank.
Have you created your database yet? Are you trying to use File/Connect, or just testing a connection in File/Configuration?
Have you tried using Windows authentication to connect? Go to the system running MSDE and login as Administrator. Open Store Operations Administrator and go to File/Connect. Enter "(local)" as the server name and select "Windows Authentication" as the login mode - leave the database name as the default and click OK. This should connect you to the database server. From here you can use the Database Menu to Create a new database or Select an existing one.
Once you verify that you can get connected to a Store Ops database using Windows authentication, you can start trying to connect using SQL Authentication.
Still from the system running SQL Server or MSDE, in Store Ops Administrator, go to File/Configuration. On the Database tab, set the server name to "(local)", the user to "sa", and the Database name to whatever name you created or connected to before. Leave the connection Timeout at it's default setting, and set the password to whatever you think it is (or leave it blank). Click "Test Database Connection".
If you don't get a successful connection like this, then you have entered the wrong sa password. IF the server is not being used for any other apps, you can change the password by again connecting using Windows authentication and running the following command in a new query window: EXEC sp_password NULL, 'theNewPassword', 'sa'
Try connecting using File/Configuration with the new password entered above.
If you can connect to the database using the sa account from the SQL server, but still have trouble from other systems, it's either a firewall problem (in the Windows XP SP2 Firewall) or the other systems can't find the server by name - give the server a static IP address and enter the IP address as the server name.