Showing posts with label encoding. Show all posts
Showing posts with label encoding. Show all posts

Wednesday, March 7, 2012

Encoding string from MS SQL

Hi all,

I have an application which will send out email in plain text in multi langauage.
the email content will be pull from txt file save in UTF-8.
i can send out email from the template with the encoding.
but when i insert data from the SQl server. the data from the SQL server are not encoded.
how do i encode the data (in other lanagauge) from sql server into UTF-8 so that it can be send together with the template.

I have try changing the data into byte and encode it in UTF-8.
but it won't displayed correctly. pls help. thanks

Unicode in SQL Server is either UCS-2 or UTF 16 and the later is the generally used version, .NET is UTF 16 by default so you can change your .NET encoding to UTF 16. The reason is NChar, NVarchar, NText, NChar max and NVarchar max are multi bytes by definition so you just need to convert to UTF 8 in your code. Hope this helps.|||

my data from the database look like this ??òú??. it is chinese simplified gb2313
the datatype for the fields is varchar.
so how do i encode it into UTF8 and it can be display in chinese.
pls help i'm totally confuse by the encoding.

|||

The Chinese you are using is Windows code page, there are six Chinese collation in SQL Server you have to find the right one in the thread below. And to UTF 8 encode in VS you start at the link below it is for VS2003 but I think it should work. So you do column level collation for your specific Chinese in SQL Server and do Unicode encoding in VS and it may be resolved. Hope this helps.

http://forums.asp.net/1067798/ShowPost.aspx

http://www.aspnetresources.com/blog/unicode_in_vsnet.aspx

|||

Hi Thanks for you advice,

base on your info i manage to find the extended proc to solve my prob xp_cp2u_web.
but i have another prob. in my stored procedure the output parameter i set it to a size of 20 data type nvarchar. it will return the result. and it will display correctly on screen. but when it is use to send via email. the rest of the content in the email is gone after my chinese character.

after some debugging, i found that it is due to the size of nvarchar i set. my chinese character size is 3. so if i set the nvarchar size to 3 all will work but if the size i set is bigger than the actual result return. it will affect the rest of the text in my email..

can anyone pls advise me on how to set the nvarchar size for my output sqlparameter.
is there a way where i can set the size of the Nvarchar as dynamic. Thanks

cmd5.CommandText = "GetEmailDetail";
cmd5.CommandType = CommandType.StoredProcedure;
cmd5.Parameters.Add("@.No", Service_ID);
SqlParameter parameterfullname = cmd5.Parameters.Add("@.name", SqlDbType.NVarChar, 20);
parameterfullname.Direction = ParameterDirection.Output;

|||

Try the link below everything you need is covered including the correct stored procedure because that is important. When you are getting value back from a SQL Server stored procedure it is OUTPUT parameter except INT which is return value, so if in doubt always use OUTPUT if it is not needed SQL Server will ignore it. So use correct column level collation and ADO.NET OUTPUT parameters. Hope this helps.

http://msdn.microsoft.com/msdnmag/issues/05/05/DataPoints/

Encoding problem

Hi,

I'm struggling to understand why I'm getting the following error when I call "sp_xml_preparedocument"

"XML parsing error: Switch from current encoding to specified encoding not supported."

The header of the xml document contains "<?xml version="1.0" encoding="UTF-8"?>"

XML data is passed into my stored procedure via an "ntext" variable so I would assume that there would'nt be an encoding issue.

Does anyone have any pointers for me? BTW I'm using SQL Server 2000.

Many thanks in advance.

Ian.

PS I've just changed the parameter from "ntext" to "text" and it works fine. Don't understand even more now.

I think ntext uses UTF-16 (respectively the older UCS-2) not UTF-8.

encoding problem

Hi
I use Oracle data source. The report is in Polish. When I try display some text from database or use report parameters, this text is not written correctly. The environment on the machine, where is installed Reporting Services is rather correct, when I am using SQL*PLUS it is ok. I linked Oracle database to MSSQL, I used Query Analyzer, the text was written correct in Polish. What to do?
morganWhat is the report language set to?
--
Ravi Mumulla (Microsoft)
SQL Server Reporting Services
This posting is provided "AS IS" with no warranties, and confers no rights.
"morgan" <morgan@.discussions.microsoft.com> wrote in message
news:9E40E53A-C040-470F-89E2-0E9807BA98D7@.microsoft.com...
> Hi
> I use Oracle data source. The report is in Polish. When I try display some
text from database or use report parameters, this text is not written
correctly. The environment on the machine, where is installed Reporting
Services is rather correct, when I am using SQL*PLUS it is ok. I linked
Oracle database to MSSQL, I used Query Analyzer, the text was written
correct in Polish. What to do?
> morgan|||Polish is the report language. By I used English ( US) too.
morgan
"Ravi Mumulla (Microsoft)" wrote:
> What is the report language set to?
> --
> Ravi Mumulla (Microsoft)
> SQL Server Reporting Services
> This posting is provided "AS IS" with no warranties, and confers no rights.
> "morgan" <morgan@.discussions.microsoft.com> wrote in message
> news:9E40E53A-C040-470F-89E2-0E9807BA98D7@.microsoft.com...
> > Hi
> >
> > I use Oracle data source. The report is in Polish. When I try display some
> text from database or use report parameters, this text is not written
> correctly. The environment on the machine, where is installed Reporting
> Services is rather correct, when I am using SQL*PLUS it is ok. I linked
> Oracle database to MSSQL, I used Query Analyzer, the text was written
> correct in Polish. What to do?
> >
> > morgan
>
>

encoding iso-8859-1

Hi Folks
to accomodate an xml file that's encoded iso-8859-1, what do I need to
change in my .xsd code (below):
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:sql="urn:schemas-microsoft-com:mapping-schema">
it looks like XMLSchema.xsd is 'wired' as utf-8. (I tried copying
XMLSchema.xsd, XMLSchema.DTD, datatypes.DTD locally and changing the encoding
in XMLSchema.xsd to iso-8859-1, but this causes other problems with simple
data type validations.)
thanks Rob
Did you add the XML declaration:
<?xml version="1.0" encoding="iso-8859-1"?>
in the beginning?
Best regards
Michael
"RobKaratzas" <RobKaratzas@.discussions.microsoft.com> wrote in message
news:1EF8BE76-F750-47D4-BA3C-6F785744C5B5@.microsoft.com...
> Hi Folks
> to accomodate an xml file that's encoded iso-8859-1, what do I need to
> change in my .xsd code (below):
> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
> xmlns:sql="urn:schemas-microsoft-com:mapping-schema">
> --
> it looks like XMLSchema.xsd is 'wired' as utf-8. (I tried copying
> XMLSchema.xsd, XMLSchema.DTD, datatypes.DTD locally and changing the
> encoding
> in XMLSchema.xsd to iso-8859-1, but this causes other problems with simple
> data type validations.)
> thanks Rob
>
|||Hi Michael
I added this to the first line (in both the .xsd AND the .XML file). The
..XML file validates just fine, however, the .xsd (which I've copied locally,
with the DTDs) comes up with:
C:\Inetpub\wwwroot\XMLSchema.xsd(1359): Restriction of anySimpleType is not
allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1359, 4).
C:\Inetpub\wwwroot\XMLSchema.xsd(1381): Restriction of anySimpleType is not
allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1381, 4).
C:\Inetpub\wwwroot\XMLSchema.xsd(1400): Restriction of anySimpleType is not
allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1400, 4).
C:\Inetpub\wwwroot\XMLSchema.xsd(1424): Restriction of anySimpleType is not
allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1424, 4).
(The SimpleType definitions fail to validate.)
For the time being, there are only several instances in the XML files where
these characters appear (I think this is more of an error with the source,
than design), so I can remove them.
But eventually, I'd like to find a solution to this.
Thanks Rob
"Michael Rys [MSFT]" wrote:

> Did you add the XML declaration:
> <?xml version="1.0" encoding="iso-8859-1"?>
> in the beginning?
> Best regards
> Michael
> "RobKaratzas" <RobKaratzas@.discussions.microsoft.com> wrote in message
> news:1EF8BE76-F750-47D4-BA3C-6F785744C5B5@.microsoft.com...
>
>
|||I'm sorry, I failed to mention these facts:
1. The validation issues come up in VS.NET 2003
2. I can browse the XMLSchema (http://localhost/XMLSchema.xsd) w/o issue
(with only inserting <?xml version="1.0" encoding="iso-8859-1"?> into the .vsd
Thanks Rob
"Michael Rys [MSFT]" wrote:

> Did you add the XML declaration:
> <?xml version="1.0" encoding="iso-8859-1"?>
> in the beginning?
> Best regards
> Michael
> "RobKaratzas" <RobKaratzas@.discussions.microsoft.com> wrote in message
> news:1EF8BE76-F750-47D4-BA3C-6F785744C5B5@.microsoft.com...
>
>
|||The error message "Restriction of anySimpleType is not
allowed" indicates that the schema contains an error. You should fix the
schema (and use the general xml newsgroup or post the offending type
restriction for more help).
HTH
Michael
"RobKaratzas" <RobKaratzas@.discussions.microsoft.com> wrote in message
news:50F3E074-06A5-4DE8-A56E-36345FD40E0F@.microsoft.com...[vbcol=seagreen]
> Hi Michael
> I added this to the first line (in both the .xsd AND the .XML file). The
> .XML file validates just fine, however, the .xsd (which I've copied
> locally,
> with the DTDs) comes up with:
> C:\Inetpub\wwwroot\XMLSchema.xsd(1359): Restriction of anySimpleType is
> not
> allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1359, 4).
> C:\Inetpub\wwwroot\XMLSchema.xsd(1381): Restriction of anySimpleType is
> not
> allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1381, 4).
> C:\Inetpub\wwwroot\XMLSchema.xsd(1400): Restriction of anySimpleType is
> not
> allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1400, 4).
>
> C:\Inetpub\wwwroot\XMLSchema.xsd(1424): Restriction of anySimpleType is
> not
> allowed. An error occurred at C:\Inetpub\wwwroot\XMLSchema.xsd, (1424, 4).
> (The SimpleType definitions fail to validate.)
> For the time being, there are only several instances in the XML files
> where
> these characters appear (I think this is more of an error with the source,
> than design), so I can remove them.
> But eventually, I'd like to find a solution to this.
> Thanks Rob
> "Michael Rys [MSFT]" wrote:

Encoding For HashBytes

When SQL Server attempts to do a MD5 hash on this string it most encode the
string to binary before hashing it. Does anyone know how varchar is encoded
,
i.e. what encoding is used for varchar? UTF-8? UTF-16? Example code:
SELECT HashBytes('MD5',CONVERT(varchar,’some string’))
If you send a nvarchar, it uses UTF-16, example:
SELECT HashBytes('MD5',CONVERT(nvarchar,’some
string’))
Thanks in advance. With this information I can write some C# code to create
a hash that matches what SQL server does.
-WayneWayne Berry (WayneBerry@.discussions.microsoft.com) writes:
> When SQL Server attempts to do a MD5 hash on this string it most encode
> the string to binary before hashing it. Does anyone know how varchar is
> encoded, i.e. what encoding is used for varchar? UTF-8? UTF-16?
> Example code:
> SELECT HashBytes('MD5',CONVERT(varchar,some string))
> If you send a nvarchar, it uses UTF-16, example:
> SELECT HashBytes('MD5',CONVERT(nvarchar,some string))
> Thanks in advance. With this information I can write some C# code to
> create a hash that matches what SQL server does.
I would suspect that it simply hashes the byte value. Which for varchar
means codes in the range 0 to 255(*), and for nvarchar a UTF-16 encoding.
(*) For Western scripts. For East Asian scripts it would be a double-
byte character set.
Erland Sommarskog, SQL Server MVP, esquel@.sommarskog.se
Books Online for SQL Server 2005 at
http://www.microsoft.com/technet/pr...oads/books.mspx
Books Online for SQL Server 2000 at
http://www.microsoft.com/sql/prodin...ions/books.mspx

Encoding errors

Somewhat complex:
I'm getting the following error in a sp (SQL Server 2000):
"Server: Msg 6603, Level 16, State 1, Procedure sp_xml_preparedocument, Line
8
XML parsing error: Switch from current encoding to specified encoding not
supported."
When a ntext xml string has the encoding set to "utf-8", but not utf-16. So
it would seem
obvious that the document stored in the database is in utf-16...?
However, when the data is written to the database, the XmlDocument class in
..NET (1.1)
is convinced the document is utf-8 (looking in the VS debugger). The call to
the sp in
C# uses:
parameter = command.Parameters.Add("@.Body", SqlDbType.NText);
parameter.Direction = ParameterDirection.Input;
parameter.Value = xd.OuterXml;
where xd is the XmlDocument. I can only conclude that somewhere the xml is
being converted
into a .NET string (and hence utf-16), I would guess at .OuterXml. But I
can't see a way
round this. At some point I need to pass the parameter value!
Any help would be appreciated.
Rgds
Peter Johnston
The problem here is that both UTF-16 and UTF-8 are encodings for UCS-4
Unicode characters but they are quite different beasts at the byte level.
SQL Server's ntext type is for historical reasons supporting UCS-2 Unicode
and for our XML parser thus should only be used to contain UCS-2 or UTF-16
encoded XML (UTF-16 uses two 2-byte code points, the so called surrogate
pairs, to represent the Unicode characters that go beyond UCS-2).
UTF-8 is a variable-length multi-byte encoding (1 to 4 bytes) of UCS-4 and
thus conflicts with the two-byte representation of ntext. The server-side
XML parser does not allow switching to UTF-8 while it parses ntext
characters since it could lead to data corruption.
Unfortunately, SQL Server does not really support a UFT-8 code page for
text/varchar either. I think that the windows-1256 encoding is mostly UTF-8
code point safe though (meaning, it does not change the code points to ?).
If you need to load data in SQL Server 2000 that contains UTF-8 encoded
data, you have the following options:
1. Change the encoding of the document to UTF-16 before passing it to the
server as ntext parameter (both the native and managed client XML libraries
have mechanisms to set the encoding)
2. If you know that you can just drop the encoding indicator, since you are
only loading ASCII anyway, do so. But be careful - even some 127-255 range
characters can get corrupted this way.
3. Load the data with the UTF-8 encoding into a text that has a collation
that implies Windows-1256.
In SQL Server 2005, we have added the ability to load the data into the XML
datatype either as XML or as varbinary() in which cases UTF-8 will work
without problems.
Best regards
Michael
"Peter Johnston" <nospam@.nospam.com> wrote in message
news:usP7DnCwFHA.2076@.TK2MSFTNGP14.phx.gbl...
> Somewhat complex:
> I'm getting the following error in a sp (SQL Server 2000):
> "Server: Msg 6603, Level 16, State 1, Procedure sp_xml_preparedocument,
> Line 8
> XML parsing error: Switch from current encoding to specified encoding not
> supported."
> When a ntext xml string has the encoding set to "utf-8", but not utf-16.
> So it would seem
> obvious that the document stored in the database is in utf-16...?
> However, when the data is written to the database, the XmlDocument class
> in .NET (1.1)
> is convinced the document is utf-8 (looking in the VS debugger). The call
> to the sp in
> C# uses:
> parameter = command.Parameters.Add("@.Body", SqlDbType.NText);
> parameter.Direction = ParameterDirection.Input;
> parameter.Value = xd.OuterXml;
> where xd is the XmlDocument. I can only conclude that somewhere the xml is
> being converted
> into a .NET string (and hence utf-16), I would guess at .OuterXml. But I
> can't see a way
> round this. At some point I need to pass the parameter value!
> Any help would be appreciated.
> Rgds
> Peter Johnston
>

Encoding errors

Somewhat complex:
I'm getting the following error in a sp (SQL Server 2000):
"Server: Msg 6603, Level 16, State 1, Procedure sp_xml_preparedocument, Line
8
XML parsing error: Switch from current encoding to specified encoding not
supported."
When a ntext xml string has the encoding set to "utf-8", but not utf-16. So
it would seem
obvious that the document stored in the database is in utf-16...?
However, when the data is written to the database, the XmlDocument class in
.NET (1.1)
is convinced the document is utf-8 (looking in the VS debugger). The call to
the sp in
C# uses:
parameter = command.Parameters.Add("@.Body", SqlDbType.NText);
parameter.Direction = ParameterDirection.Input;
parameter.Value = xd.OuterXml;
where xd is the XmlDocument. I can only conclude that somewhere the xml is
being converted
into a .NET string (and hence utf-16), I would guess at .OuterXml. But I
can't see a way
round this. At some point I need to pass the parameter value!
Any help would be appreciated.
Rgds
Peter JohnstonThe problem here is that both UTF-16 and UTF-8 are encodings for UCS-4
Unicode characters but they are quite different beasts at the byte level.
SQL Server's ntext type is for historical reasons supporting UCS-2 Unicode
and for our XML parser thus should only be used to contain UCS-2 or UTF-16
encoded XML (UTF-16 uses two 2-byte code points, the so called surrogate
pairs, to represent the Unicode characters that go beyond UCS-2).
UTF-8 is a variable-length multi-byte encoding (1 to 4 bytes) of UCS-4 and
thus conflicts with the two-byte representation of ntext. The server-side
XML parser does not allow switching to UTF-8 while it parses ntext
characters since it could lead to data corruption.
Unfortunately, SQL Server does not really support a UFT-8 code page for
text/varchar either. I think that the windows-1256 encoding is mostly UTF-8
code point safe though (meaning, it does not change the code points to ?).
If you need to load data in SQL Server 2000 that contains UTF-8 encoded
data, you have the following options:
1. Change the encoding of the document to UTF-16 before passing it to the
server as ntext parameter (both the native and managed client XML libraries
have mechanisms to set the encoding)
2. If you know that you can just drop the encoding indicator, since you are
only loading ASCII anyway, do so. But be careful - even some 127-255 range
characters can get corrupted this way.
3. Load the data with the UTF-8 encoding into a text that has a collation
that implies Windows-1256.
In SQL Server 2005, we have added the ability to load the data into the XML
datatype either as XML or as varbinary() in which cases UTF-8 will work
without problems.
Best regards
Michael
"Peter Johnston" <nospam@.nospam.com> wrote in message
news:usP7DnCwFHA.2076@.TK2MSFTNGP14.phx.gbl...
> Somewhat complex:
> I'm getting the following error in a sp (SQL Server 2000):
> "Server: Msg 6603, Level 16, State 1, Procedure sp_xml_preparedocument,
> Line 8
> XML parsing error: Switch from current encoding to specified encoding not
> supported."
> When a ntext xml string has the encoding set to "utf-8", but not utf-16.
> So it would seem
> obvious that the document stored in the database is in utf-16...?
> However, when the data is written to the database, the XmlDocument class
> in .NET (1.1)
> is convinced the document is utf-8 (looking in the VS debugger). The call
> to the sp in
> C# uses:
> parameter = command.Parameters.Add("@.Body", SqlDbType.NText);
> parameter.Direction = ParameterDirection.Input;
> parameter.Value = xd.OuterXml;
> where xd is the XmlDocument. I can only conclude that somewhere the xml is
> being converted
> into a .NET string (and hence utf-16), I would guess at .OuterXml. But I
> can't see a way
> round this. At some point I need to pass the parameter value!
> Any help would be appreciated.
> Rgds
> Peter Johnston
>