Your current location:

Evaluation method


1. evaluation process

First of all, starting from the definition of the connotation of the science and technology community, combined with the world-class science and technology community case analysis and structure-process-results (SPO) theoretical model, the initial formation of the world-class science and technology community evaluation content framework.

Secondly, combing the existing evaluation indicators, holding a seminar to invite experts to score, using AHP level analysis to determine the weight of each indicator, to determine the evaluation index system.

Again, according to the evaluation index system to collect relevant data, data cleaning and conversion, after data integration calculation, the formation of evaluation results.

 

2. index system design

(1) Design principles

a) Scientific principle. The design of the evaluation system of science and technology associations should be based on the relevant theoretical research and expert wisdom, fully reflect the actual development law of science and technology associations, and fit the nature and characteristics of the associations.

B) Systematic principle. According to the system theory, the design of the evaluation system should pay attention to the integrity and comprehensiveness, on the one hand, we should consider the overall characteristics of the scientific and technological community from internal characteristics to external performance, from basic differences to development trends, on the other hand, the selection of indicators should be exhaustive and mutually exclusive, considering the internal connection between the various elements.

c) Guiding principles. The evaluation index system should reflect the value orientation and key characteristics of scientific and technological associations. Through the implementation of evaluation, the results should be able to guide scientific and technological associations to improve their own problems to enhance their comprehensive ability.

d) Principle of feasibility. The evaluation system should be concise and operable, the data source should be stable, open and available, and the evaluation results should be of practical significance.

e) the principle of comparability. The index system should try to include the common attributes and activities of scientific and technological associations, and should not lead to the deviation of evaluation results due to personalized differences. Try to use objective and public data to reduce the possibility of substituting subjective will in the evaluation.

(2) Evaluation content

"The evaluation index system of world-class science and technology associations" mainly evaluates science and technology associations from three levels of organization, cohesion and influence.

Organizational power (Organizing Power): refers to the ability to design organizational structure and allocate organizational resources, is the internal strength of the organization's self-growth, is to maintain competitive advantage and sustainable development of the power source. Internal governance reflects the organization and leadership ability and system guarantee of the association, and reflects the resource level of human, capital and information of the science and technology association at this stage.

Cohesion (Integrating Power): refers to the ability of the science and technology community to attract each other, relate to each other and identify with common goals through specific structures and activities. It is the cohesion and attraction as the core of the organization. According to the main activities and functions of the members of the science and technology community, it can be divided into five dimensions: historical inheritance, talent attraction, knowledge generation, science popularization and policy advocacy.

Impact Power (Impact Power): refers to the direct performance of the knowledge construction and dissemination ability of the science and technology community, which reflects the effect of the organization and cohesion of the science and technology community, and is an important basis for the social credibility of the science and technology community. Focus on the reputation and influence of different scopes such as science, society, and international, and measure it by selecting indicators such as the level of representative journals, network communication and international exchanges.

 

Table 1 Evaluation index system of world-class science and technology associations (2019 trial version)

 

Level 1 indicators
(3)
Secondary indicators
(10)
Level 3 index
(18)
Measurement index and description Data source
organizational force
(30%)
Internal governance
(40%)
division of labor and cooperation Clear organizational structure, clear division of labor among directors, supervisors and secretary-general, able to effectively play decision-making and supervisory functions Association official website, annual report, Gale database, Wiki encyclopedia, national tax websites, statistical yearbook of the Chinese Association for Science and Technology.
institutional norms The degree of standardization of the system charter publicly displayed on the association website (purpose vision, board election, supervision, regular office management, board meeting, member membership management, professional committee, working committee, financial regulations, ethics) is assigned in ten aspects.
Basic guarantee
(60%)
Human Resources Number of individual members
Funding resources Total Capital Income in 2017
information resources Whether financial information is made public, whether annual reports are published; the number of official social media
Cohesion
(30%)
historical inheritance
(5%)
Time of establishment Number of years since the association was formally established until 2018 Association official website, annual report
Talent Attraction
(30%)
Member Services Number of various service items set for members
science and technology award Number of awards set by associations for discipline development and talent training
knowledge creation
(30%)
periodical platform Number of academic journals sponsored by associations
Conference Exchange Number of various conferences held in recent three years
science popularization
(30%)
Scientific Publications Total number of all kinds of traditional or electronic publications published by associations for knowledge popularization in the last three years
education and training Number of online or field education courses and activities held in the past three years, number of subject competitions organized
Policy advocacy
(5%)
Public Statement In the past three years, the association has issued a formal statement on a major event through the public media, submitted suggestions to government departments, open letters, etc.
Influence
(40%)
Academic Impact
(40%)
innovation potential Growth rate of impact factors in recent five years ISI Database
Academic lead The number of journals sponsored by the association and included by SCI and SSCI, which are ranked in various disciplines, and the impact factor
Social impact
(35%)
network propagation Global ranking of community website page views Alexa's website
Social Media The number of social media fans in Twitter, Weibo and other social media Twitter, Sina Weibo
International Impact (25%) International cooperation Number of international partner countries listed on the website, number of countries where international branches are located Community official website

 

Research and literature analysis found that first-class academic journals are one of the typical characteristics of world-class scientific and technological associations, so this study takes the world-recognized first-class academic journals as a necessary condition for finding world-class scientific and technological associations. The selection of evaluation objects is based on the 2018 SCI and SSCI periodical databases published by Clarivate companies, and nearly 2000 world-class scientific and technological associations are sorted out. On this basis, management experts and scientific and technological workers of scientific and technological associations are further invited to recommend 89 scientific and technological associations. According to the sample selection criteria of world-class, transparency, comparability and stability, an alternative pool of evaluation objects of world-class scientific and technological associations is formed. Based on the consistency of the evaluation objects, this evaluation focuses on academic associations, excluding international/regional, comprehensive, federations and other scientific and technological associations. Therefore, this study only selects about 500 academic scientific and technological associations with relatively high academic level, wide audience, relatively available and accurate other statistical data as evaluation objects.

The data sources mainly include three channels: first, the official websites of science and technology organizations to learn about annual reports, financial reports, activities and other information; Second, the official statistical publications of various governments, such as the US 990 tax table and the statistical yearbook of the China Association for Science and Technology. Third, internationally renowned databases, including ISI, SCOPUS, Gale Associations Unlimited, etc., as well as public Internet websites such as ALEXA, Twitter, Sina Weibo, Sina Weibo, etc. The statistical yearbook mainly uses 2017 data, the network data is mainly in 2018, and the number of some activities is the average value of 2016-2018.

The data acquisition is carried out by the method of crawler program extraction, manual identification and verification. Raw information related to the indicators is first collected through the Internet and then quantified according to the relevant formulas.

 

3. data calculation method

The evaluation calculation of 500 scientific and technological societies in the world uses the equal-weighting method from three levels to two levels, and then from the second level to the second level. Different methods are used in the calculation of each indicator: first, direct percentile processing, such as rules and regulations, and second, the use of normal distribution as the core to rank-based scoring method, the original numerical standardization processing to calculate the z value, and then calculate the normal distribution probability value P(X = z), finally, multiply the P value by 100 to obtain the final percentile score such as journal platform, conference paper, international branch, network communication, etc. Third, the comprehensive index calculation method, such as excellent journals using HIF and sumRank two indicators set equations to solve.